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Executive  Summary 


1.0  Overview 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  was 
chartered  by  the  Deputy  Director,  Test,  Systems  Engineering,  and  Evaluation1  (Test  and 
Evaluation),  Office  of  the  Secretary  of  Defense  (Acquisition  and  Technology)  in  October  1994  to 
investigate  the  utility  of  advanced  distributed  simulation  (ADS)  technologies  for  support  of 
distributed  testing  in  development  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation 
(OT&E).  In  keeping  with  the  JADS  charter  to  explore  the  utility  of  ADS  for  test  and  evaluation 
(T&E),  this  special  report  provides  insight  into  the  benefits  and  costs  associated  with  the  use  of 
distributed  testing  in  developmental  and  operational  testing.  It  provides  program  managers  (PMs) 
with  findings,  conclusions,  and  lessons  learned  from  the  three  JADS  tests  and  other  agencies 
regarding  the  costs  and  benefits  of  using  distributed  testing.  It  is  the  combined  effort  of  the  JADS 
Joint  Test  Force  (JTF)  and  the  MITRE  Corporation’s  Economic  and  Decision  Analysis  Center 
(EDAC)  and  consists  of  two  sections.  The  first  part  describes  the  potential  benefits  and  cost 
savings  arising  from  the  incorporation  of  distributed  testing.  The  second  half  introduces  a  work 
breakdown  structure  (WBS)  format  to  support  the  assessment  of  the  costs  and  the  risks  of 
incorporating  distributed  testing  into  the  T&E  process.  Additionally,  the  second  part  identifies 
cost  drivers  and  areas  of  potential  risk.  The  two  appendices  present  case  studies  where  the  results 
of  previously  conducted  actual  live  tests  were  compared  to  the  notional  results  one  might  have 
obtained  had  the  original  test  been  enhanced  with  distributed  testing.  Distributed  testing 
technologies  provide  new  capabilities  that  potentially  help  the  tester  overcome  limitations  with 
traditional  testing.  This  report  has  generated  several  recommendations  for  the  Department  of 
Defense  (DoD)  T&E  community.  These  recommendations  focus  on  the  global  T&E  community’s 
needs  and,  if  followed,  will  help  PMs  to  better  conduct  distributed  testing. 

Much  of  the  discussion  concerning  distributed  testing  for  T&E  infers  that  distributed  testing  is 
separate  from  other  more  traditional  means  of  testing.  Sometimes  the  use  of  distributed  testing  is 
described  as  an  “alternative”  to  traditional  testing,  and  costs  of  a  test  program,  with  and  without 
distributed  testing,  are  compared.  Distributed  testing  can  be  thought  of  more  precisely  as  a 
technique  for  bringing  multiple,  physically  separated  data  sources  together  in  a  shared  interactive 
environment.  It  is  simply  another  (albeit  poorly  understood)  tool  available  to  the  test  designer. 
The  WBS  highlights  where  distributed  testing  uniquely  impacts  the  traditional  test  WBS.  Once 
distributed  testing  becomes  more  common  place,  the  unique  elements  of  this  WBS  will  either 
become  standard  or  will  be  absorbed  into  the  definitions  of  the  next  higher  elements. 


1  This  office  is  now  the  Deputy  Director,  Developmental  Test  and  Evaluation,  Strategic  and 
Tactical  Systems. 
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2.0  Characterizing  the  Benefits  of  Distributed  Testing 

The  structure  of  an  optimal  testing  program  should  be  based  on  the  appropriate  balance  between 
cost  savings  and  benefits.  To  determine  the  benefits  of  any  new  capability  or  technology  there 
must  be  objective  standards  against  which  the  performance  of  the  new  capability  or  technology  is 
measured.  These  performance  measures  or  standards  can  be  both  quantitative  and  qualitative. 
This  report  categorizes  performance  or  effectiveness  measures  as  either  standards  related  to 
enhanced  testing  capability  (is  it  better  or  faster?)  or  to  standards  related  to  cost  reduction 
potential  (is  it  cheaper?). 

In  deciding  whether  to  use  distributed  testing,  the  tester  should  first  determine  its  effectiveness. 
To  do  this  the  tester  would  first  estimate  the  potential  benefits  of  implementing  distributed  testing 
and  determine  any  testing  constraints  that  might  limit  its  implementation.  After  identifying 
potential  benefits  and  constraints,  the  PM  would  estimate  the  costs  with  and  without  distributed 
testing.  Balancing  the  benefits  and  costs  would  allow  the  structure  of  an  optimal/near  optimal 
distributed  testing  program.  Any  feasibility  analysis  should  provide  information  on  when,  where, 
and  how  distributed  testing  methods  can  best  be  applied  to  complement  traditional  test  methods 
and  enhance  the  overall  T&E  phase.  The  JADS  experience  has  shown  that  distributed  testing 
technologies  can  complement  other  T&E  approaches  but  should  not  necessarily  replace  more 
traditional  forms  of  testing  (e.g.,  live  testing). 

To  illustrate  the  decision  process,  this  report  outlines  three  possible  approaches  to  the  application 
of  distributed  testing:  (1)  a  cost  savings  approach,  (2)  a  cost  neutral  approach  and  (3)  a  more 
effective  testing  approach.  In  the  first  approach,  substituting  distributed  testing  for  selected  live 
testing  reduces  total  testing  costs.  In  the  second  case,  distributed  testing  is  added  and  the  scope 
of  other  testing  is  reduced.  The  third  approach  adds  distributed  testing  without  eliminating  any 
live  testing,  so  that  total  costs  are  higher  but  with  the  benefit  of  improved  testing. 

3.0  Cost  Guidance 

The  second  half  of  the  report,  written  by  MITRE  Corporation’s  EDAC  and  supported  by  JADS, 
provides  cost  guidance  to  help  the  PM  make  cost  estimates.  This  section  provides  information 
that  T&E  organizations  may  find  useful  in  costing  testing  methods  and  processes. 

The  cost  guidance  is  based  on  the  JADS  test  experience  and  is  presented  at  a  level  above  a  cost 
model.  As  a  result,  it  is  a  useful  first  step  in  supporting  a  given  program’s  distributed  testing  cost 
estimating  efforts.  For  example,  the  cost  guidance  addresses  cost  drivers,  major  areas  of  risk,  risk 
mitigation  and  lessons  learned.  Although  the  cost  guidance  falls  short  in  providing  a  complete 
estimate  of  distributed  testing  costs  for  every  specific  requirement,  it  will  help  PMs  begin  to 
determine  the  costs  of  incorporating  distributed  testing  into  their  T&E  program. 
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1.0  Overview 


The  purpose  of  this  special  report  is  to  provide  insight  into  the  benefits  and  costs  associated  with 
the  use  of  advanced  distributed  simulation  (ADS)  in  developmental  and  operational  testing.  It  is 
the  combined  effort  of  the  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  Force  (JTF) 
and  the  MITRE  Corporation’s  Economic  and  Decision  Analysis  Center  (EDAC)  and  consists  of 
two  sections.  The  first  section  discusses  the  overall  potential  benefits  and  cost  savings  associated 
with  the  use  of  distributed  testing  for  test  and  evaluation  (T&E).  The  second  provides  cost 
guidance  for  incorporating  distributed  testing  into  the  T&E  phase  of  a  developmental  program 
and  structures  information  collected  from  the  three  JADS  tests,  as  well  as  information  from  other 
agencies  using  distributed  testing,  into  a  work  breakdown  structure  (WBS),  a  common  program 
management  format.  The  two  appendices  present  case  studies  where  the  results  of  previously 
conducted  actual  live  tests  were  compared  to  the  notional  results  one  might  have  obtained  had  the 
original  test  been  enhanced  with  distributed  testing. 

1.1  Background  and  Assumptions 

This  report  relies  on  JADS  findings,  results,  and  lessons  learned.  The  Office  of  the  Secretary  of 
Defense  chartered  JADS  in  October  1994  to  investigate  the  utility  of  distributed  testing  for  T&E. 
The  JADS  program  consists  of  three  multiphased  test  programs:  the  System  Integration  Test 
(SIT)  explored  distributed  testing  of  precision  guided  munitions  (PGM);  the  End-to-End  (ETE) 
Test  investigated  distributed  testing  of  command,  control,  communications,  computers, 
intelligence,  surveillance  and  reconnaissance  (C4ISR)  systems;  and  the  Electronic  Warfare  (EW) 
Test  examined  distributed  testing  of  EW  systems.  For  further  information  on  these  tests  consult 
the  JADS  website  at  http://www.iads.abq.com/html/iads/iads.htm.2 

Much  of  the  discussion  concerning  distributed  testing  for  T&E  infers  that  distributed  testing  is 
separate  from  other  more  traditional  means  of  testing.  Sometimes  the  use  of  distributed  testing  is 
described  as  an  “alternative”  to  traditional  testing,  and  costs  of  a  test  program,  with  and  without 
distributed  testing,  are  compared.  Distributed  testing  can  be  thought  of  more  precisely  as  a 
technique  for  bringing  multiple,  physically  separated  data  sources  together  in  a  shared  or  real 
devices  or  people.  The  virtual  battle  space  or  test  space  created  by  the  interactions  of  the  data 
sources  is  indeed  a  “model”  because  the  virtual  space  is  a  representation  of  reality.  It  is  not, 
however,  a  model  in  the  traditional  sense.  It  is  not  lines  of  code  being  processed  in  a  single 
machine  or  multiple  machines  linked  together.  Within  distributed  testing,  it  is  feasible  to  have  a 
cast  of  players  who  are  all  “real”  —  perhaps  even  operating  in  a  real  environment.  In  such  a  case, 
it  is  possible  to  generate  and  collect  traditional  test  data.  That  is  very  different  from  a  stand-alone 
model  where  actual  test  data  must  be  put  into  the  model.  Even  in  cases  where  live,  virtual,  and 
constructive  data  sources  are  mixed,  the  system  under  test  (SUT)  may  be  the  real  thing  and  the 


2  After  1  March  2001  refer  requests  to  Headquarters  Air  Force  Operational  Test  and  Evaluation 
Center  History  Office  (HQ  AFOTEC/HO),  8500  Gibson  Blvd.  SE,  Rutland  Air  Force  Base,  New 
Mexico  87117-5558,  or  Science  Applications  International  Corporation  (SAIC)  Technical 
Library,  2001  North  Beauregard  St.,  Suite  80,  Alexandria,  Virginia  22311. 
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virtual  environment  can  supply  measurable  stimuli  for  testing.  The  key  difference  between  a 
distributed  testing  architecture  “model”  and  a  constructive  “model”  is  the  direction  of  flow  of  test 
information.  In  the  distributed  testing  case,  the  flow  can  be  from  the  SUT  to  the  analyst.  In  the 
constructive  case,  the  analyst  must  incorporate  test  data  into  the  model,  and  the  lines  of  code 
generate  output  for  the  analyst.  In  the  latter  case  the  output  might  be  thought  of  as  the  analytical 
conclusion  of  the  model.  If  the  analyst  doesn’t  like  or  believe  the  output  he  or  she  can  manipulate 
input  data,  model  parameters,  or  algorithms  to  “adjust”  the  answers.  The  constructive  model  may 
yield  a  variety  of  insights,  but  it  does  not  provide  “test”  data  as  the  distributed  testing 
environment  can.  Additionally,  where  human  beings  are  important  elements  of  testing,  the 
distributed  testing  environment  supports  their  participation.  Constructive  models  have  historically 
had  trouble  realistically  representing  human  processes. 

1.2  The  Use  of  Distributed  Testing  as  Part  of  the  Total  Acquisition  Process 

The  use  of  distributed  testing  technologies  is  not  limited  to  operational  test  and  evaluation 
(OT&E)  and  developmental  test  and  evaluation  (DT&E).  The  technology  can  support  the  other 
testing  phases:  early  operational  assessment  (EOA),  operational  assessment  (OA),  initial 
operational  test  and  evaluation  (IOT&E),  and  follow-on  operational  test  and  evaluation  (FOT&E) 
of  an  overall  acquisition  program. 

For  example,  Figure  1  illustrates  the  role  of  distributed  testing  during  a  precision  guided  missile 
(PGM)  system  acquisition  and  testing  life  cycle.  Similar  diagrams  can  be  constructed  for  most 
system  development  efforts.  A  PGM  digital  system  model  (DSM)  normally  becomes  available 
during  the  initial  system  acquisition  phase,  so  that  evaluations  can  begin  during  this  phase  (e.g., 
for  requirements  development),  even  before  formal  T&E  begins.  The  optimal  use  of  the  various 
methods  depends  on  the  PGM  performance  areas  being  evaluated.  Note  from  Figure  1  that 
distributed  testing  can  be  used  throughout  the  system  acquisition  and  testing  life  cycle  as  PGM 
simulation  resources  are  developed  and  refmed.  Although  similar  illustrations  are  not  presented 
for  EW  and  C4ISR  systems,  distributed  testing  can  be  used  throughout  the  acquisition  and  testing 
life  cycles  of  those  types  of  systems  as  well. 

DSMs  are  expected  to  be  more  common  as  Simulation  Based  Acquisition  takes  hold.  While 
JADS  did  not  use  DSMs  in  the  PGM  test,  JADS  did  use  a  DSM  in  the  EW  Test  and  in  the  C4ISR 
test.  In  the  EW  Test,  JADS  used  a  DSM  of  an  airborne  self-protection  jammer  interacting  with 
manned  threat  simulators  in  a  geographically  separate  facility  to  recreate  an  open  air  test.  The 
DSM  in  this  test  represented  a  very  early  system  model  capable  only  of  simulating  the  jammer 
logic  and  decision  cycle  times.  This  shows  how  T&E  methods  can  be  brought  to  bear  as  soon  as 
there  are  system  components  or  subcomponents  available,  even  if  the  system  component  is  only 
the  DSM.  As  system  acquisition  proceeds,  broader  ranges  of  T&E  methods  can  continue  to  be 
used  to  test  the  evolving  system  and  subsystems.  The  JADS  C4ISR  test  used  a  more  evolved 
type  of  DSM  constmcted  primarily  from  the  systems  operational  software  coupled  with  models  of 
the  platform  sensor.  The  resulting  DSM  was  suitable  for  system  OT&E,  training,  and  future 
operational  flight  program  (OFP)  development. 
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DSM  =  digital  system  mode! 


HWIL  =  hardware-in-the-loop  MS  =  milestone 


Figure  1.  Role  of  Distributed  Testing  During  PGM  System  Acquisition 

and  Testing  Life  Cycle 

1.3  Benefits  of  Implementing  Distributed  Testing 

The  structure  of  an  optimal  testing  program  should  be  based  on  the  appropriate  balance  between 
cost  savings  and  benefits.  To  determine  the  benefits  of  any  new  capability  or  technology  there 
must  be  objective  standards  against  which  the  performance  of  the  new  capability  or  technology  is 
measured.  These  performance  measures  or  standards  can  be  both  quantitative  and  qualitative. 
This  report  categorizes  performance  or  effectiveness  measures  as  either  standards  related  to 
enhanced  testing  capability  (is  it  better  or  faster?)  or  to  standards  related  to  cost  reduction 
potential  (is  it  cheaper?). 

1.3.1  Enhanced  Test  Capability 

JADS  demonstrated  that  distributed  testing  can  overcome  many  of  the  traditional  limitations  and 
problems  with  test  and  evaluation.  Distributed  testing  allows  a  richer,  more  reactive  environment 
to  be  created  earlier  in  system  development.  Traditional  single  model  or  model  versus  model 
analysis  is  not  as  reactive  as  simulations  where  human  intelligence  is  allowed  to  affect  appropriate 
system  actions.  Human  operators  are  an  integral  part  of  many  weapon  systems  and  need  to  be 
part  of  early  system  testing  if  possible.  For  example,  using  models  to  determine  jammer 
effectiveness  against  manned  threats  ignores  the  human  operator’s  ability  to  recognize  the  target 
in  the  jammed  display  increasing  the  risk  that  the  jammer  will  be  ineffective.  By  allowing  the 
digital  model  to  interact  with  the  manned  threat  simulators,  distributed  testing  allows  the  system 
developer  to  reduce  the  development  risk  by  measuring  jammer  effectiveness  early  on.  In  the 


5 


PGM  example,  linked  laboratories  could  provide  reproducible,  higher  confidence  results.  Missile 
testing  using  a  linked  laboratory  distributed  testing  architecture  is  more  reproducible  than  live 
testing  because  scenario  conditions  are  more  readily  controlled  and  trials  can  be  replayed  for 
additional  PGM  responses.  This  allows  more  trials  to  be  combined  for  analysis,  giving  greater 
confidence  in  evaluation  results.  Distributed  testing  also  injects  more  realism  than  analytical 
models  since  actual  hardware  is  used,  and  linked  simulation  is  often  more  realistic  than  stand¬ 
alone  hardware-in-the-loop  (HWIL)  laboratories.  Distributed  testing  allows  the  test  designer  to 
take  advantage  of  the  laboratory’s  inherent  abilities  to  provide  secure  evaluation  of  classified 
electronic  countermeasures  (ECM)  techniques  and  to  increase  force  density  or  representation 
through  the  use  of  simulation.  In  the  C4ISR  arena,  distributed  testing  allows  the  force  density  of 
the  scenario  to  be  increased  affordably.  The  number  of  friendly  and  threat  systems  can  be 
increased  by  representing  them  with  either  manned  laboratories  (if  realistic  man-in-the-loop 
control  of  the  systems  is  needed)  or  DSMs  (if  scripted  behavior  is  acceptable).  The  inability  to 
evaluate  system  performance  in  combat-representative  environments  is  a  common  limitation  in 
OT&E  and  an  area  in  which  distributed  testing  can  improve  the  operational  test  (OT) 
environment.  The  ability  of  distributed  testing  to  create  affordable,  large-scale  and  complex 
environments  for  the  SUT  could  mean  more  thorough  testing.  That,  in  turn,  could  provide  early 
identification  of  problems  that  might  otherwise  go  unnoticed.  There  is  also  a  potential  for 
reducing  test  duration  by  using  multiple  facilities  in  an  integrated  environment  rather  than  using 
them  sequentially. 

JADS  found  utility  for  distributed  testing  in  each  of  the  areas  investigated.  Key  areas  of  utility  are 
described  below. 

1.3. 1.1  Precision  Guided  Munitions  Enhanced  Test  Capability 

The  SIT  investigated  the  ability  of  distributed  testing  to  support  air-to-air  missile  testing.  The  test 
included  two  sequential  phases,  a  Linked  Simulators  Phase  (LSP)  and  a  Live  Fly  Phase  (LFP). 
Both  phases  incorporated  one-versus-one  scenarios  based  upon  profiles  flown  during  live  test 
activities  and  limited  target  countermeasure  capability. 

The  LSP  distributed  architecture  incorporated  four  nodes:  the  shooter,  an  F/A-18  manned 
avionics  laboratory  at  China  Lake,  California;  the  target,  an  F-14  manned  avionics  laboratory  at 
Point  Mugu,  California;  a  HWIL  missile  laboratory  at  China  Lake  which  hosted  an  air  intercept 
missile  (AIM)-9M  missile;  and  a  test  control  center  initially  located  at  Point  Mugu  and  later 
relocated  in  the  JADS  facility  in  Albuquerque,  New  Mexico. 

The  LFP  distributed  architecture  linked  two  live  F-16  aircraft  (a  shooter  and  target)  on  the  Eglin 
Air  Force  Base,  Florida,  Gulf  Test  Range;  the  Eglin  Central  Control  Facility;  an  HWIL  missile 
laboratory  at  Eglin  which  hosted  an  AIM- 120  missile;  and  a  test  monitoring  center  at  the  JADS 
facility  in  New  Mexico. 

This  report  describes  the  outcome  of  the  SIT,  the  conclusions  and  lessons  learned,  and  offers 
observations  on  the  implications  of  SIT  for  the  general  class  of  precision  guided  munitions. 
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1 . 3 . 1 . 1 . 1  System  Integration  Test  Results  and  Conclusions 

Within  the  narrow  confines  of  the  SIT  data,  our  assessment  is  that  the  two  architectures  we 
employed  have  utility  for  support  of  T&E.  The  JADS  data  indicate  that  activities  ranging  from 
parametric  analyses  to  integrated  weapons  system  testing  are  both  practical  and  cost  effective. 
Our  broad  conclusions  and  lessons  learned  can  be  summarized  as  follows. 

-  For  T&E  applications,  the  technology  is  not  at  the  “plug-and-play”  stage.  While  practical  and 
cost  effective  in  many  cases,  implementation  is  more  challenging  than  many  people  think. 
Plan  for  a  lot  of  rehearsals  and  “fix”  time. 

-  The  effects  of  latency  and  other  ADS-induced  errors  can  often  (not  always)  be  mitigated. 

-  Synchronization  is  as  much  a  challenge  as  latency. 

-  Instrumentation  and  data  management  are  challenges. 

-  ADS  has  great  potential  as  a  T&E  support  tool;  it  is  a  valuable  addition  to  the  tester’s  tool 
kit.  ADS  will  not  obviate,  but  in  some  cases  it  may  reduce,  the  need  for  live  testing. 

-  Our  data  suggest  test  savings  are  possible. 

1 .3 . 1 . 1 .2  Observations  for  Precision  Guided  Munitions  T&E 

As  JADS  demonstrated  in  the  System  Integration  Test,  the  live  shooter-target  distributed  testing 
architecture  can  be  used  for  realistic  one-versus-one  engagement  tactics  development  and 
evaluation.  Distributed  testing  exhibits  more  realism  than  either  analytical  simulation  models 
(because  actual  hardware  is  used)  or  stand-alone  HWIL  laboratories  (because  realistic  shooter 
and  target  inputs  are  provided). 

Shooter/PGM  integration  can  be  effectively  evaluated  using  distributed  testing  configurations. 
Integration  check-out  can  begin  early  in  the  acquisition  cycle  before  a  complete  PGM  system  is 
available  by  using  the  linked  laboratory  distributed  testing  configuration  and  including  only  key 
PGM  subsystems  in  the  PGM  HWIL  laboratory.  Note  that  this  same  configuration  can  also  be 
used  to  verify  the  integration  of  a  new  shooter  platform  into  an  existing  PGM  system.  If  the  PGM 
receives  data  link  support  from  the  shooter  during  its  flyout,  the  live  shooter-target  distributed 
testing  configuration  can  be  used  to  accurately  evaluate  the  quality  of  the  support  messages  and 
the  resulting  interactions. 

Testing  using  a  live  shooter-target  distributed  testing  architecture  is  more  efficient  than  live  shot 
testing  because  the  analysts  get  immediate  feedback  on  each  trial  of  a  multiple  trial  mission.  This 
allows  adjustments  to  be  made  to  the  remaining  test  matrix,  if  necessary,  while  the  live  shooter 
and  target  platforms  are  still  on  range.  This  “analyst-in-the-loop”  feature  of  distributed  testing 
would  be  especially  useful  in  efficiently  progressing  through  an  ECM  testing  matrix  which 
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involves  varying  a  number  of  ECM-related  parameters.  (Up  to  15  trials  were  executed  during 
each  two-hour  SIT  LFP  mission.) 

Live  fire  tests  can  be  realistically  rehearsed  using  distributed  testing.  This  would  ensure  the 
proper  setup  of  the  scenario  and  reduce  wasted  live  fire  attempts  in  which  the  proper  scenario 
conditions  are  not  achieved  or  would  result  in  anomalies  (i.e.,  cost  avoidance).  This  use  of 
distributed  testing  would  also  reduce  the  risk  of  a  live  fire  testing  program  by  identifying  scenarios 
that  cannot  be  correctly  executed  or  which  cannot  achieve  the  stated  objectives. 

Also,  distributed  testing  implementation  may  benefit  other  parts  of  the  PGM  acquisition  program 
besides  testing.  Distributed  testing  using  a  DSM  to  represent  the  PGM  may  aid  in  requirements 
development,  and  resources  developed  during  distributed  testing  implementation  may  be  useful 
for  training.  Such  benefits  of  distributed  testing  implementation  that  are  beyond  the  normal  scope 
of  testing  should  also  be  considered  in  this  determination.  The  distributed  testing  architecture 
also  can  support  real-time  distribution  of  test  results  to  appropriate  analytical  agencies  not 
involved  in  testing.  There  may  be  significant  efficiencies  in  planning  for  future  test  events  or 
assessing  the  need  for  system  improvements. 

Through  a  process  of  inductive  reasoning  we  can  transfer  some  of  the  SIT-based  specifics  to  the 
general  class  of  PGM.  In  the  general  case,  the  elements  of  the  SIT  architectures  are  basic  to  all 
PGM  cases.  There  are  (1)  a  launch  platform  or  shooter,  (2)  a  PGM,  (3)  an  intended  target,  (4)  an 
operating  environment  (to  include  countermeasures),  and  (5)  a  test  control  center. 

The  shooter,  PGM,  and  target  can  be  represented  in  any  of  the  three  forms  associated  with 
distributed  simulation:  live,  virtual,  or  constructive.  SIT  looked  at  an  AIM-9  and  an  AIM-120. 
The  physical  dynamics  of  the  problem  are  comparable  with  any  class  of  PGM.  The  physics 
associated  with  detection,  tracking,  and  guidance  may  differ  significantly  depending  upon  bands, 
techniques,  and  the  operational  medium  a  missile  operates  in.  We  do  not  see  a  one-for-one 
transfer  of  SIT  techniques  to  other  tests.  Each  test  has  specific  requirements,  often  peculiar  to  the 
particular  system  under  test.  We  do  see  a  transfer  of  the  principles,  design  processes,  and 
methodologies  used  in  SIT. 

Testing  using  manned  shooter  and  target  simulators  linked  to  a  PGM  DSM  or  HWIL  laboratory 
can  be  used  to  realistically  evaluate  man-in-the-loop  reactive  countermeasures  (CM).  This  cannot 
be  done  in  live  fire  testing  because  of  obvious  safety  constraints.  It  is  also  possible  to  create 
launch  profiles  that  examine  missile  performance  under  conditions  too  dangerous  to  fly.  CM  were 
only  represented  in  rudimentary  form  in  the  SIT,  but  we  see  no  technical  impediments,  at  the 
conceptual  level,  to  implementing  high-fidelity  countermeasures  in  distributed  testing.  The  actual 
costs  and  technical  challenges  will  be  very  case  specific.  Complex  environmental  details 
associated  with  atmospherics,  space,  oceanography,  etc.,  are  more  challenging.  In  the  SIT  case, 
the  LFP  incorporated  real  atmospheric  effects. 

A  test  control  center  is  a  requirement  for  all  testing,  distributed  or  not.  Fortunately,  the  SIT 
experience  suggests  that  the  control  center  can  function  from  almost  anywhere.  The  inference  is 
that  an  existing  control  center  somewhere  may  well  meet  a  specific  tester’s  needs. 
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The  SIT  program  was  budget  and  schedule  constrained.  Consequently,  there  were  important 
aspects  of  PGM  testing  which  SIT  did  not  explore.  From  a  single  shooter  perspective,  some  of 
these  included  multiple  launches  against  a  single  target,  single  launches  against  clustered  targets, 
and  multiple  launches  against  multiple  targets.  SIT  did  not  examine  few-on-few  or  many-on- 
many  scenarios.  Our  expectation,  unsupported  by  hard  data,  is  that  few-on-few  implementations 
are  possible.  The  difficulties  and  costs  would  be  extremely  sensitive  to  the  fidelity  requirements 
and  the  availability  of  existing  facilities,  e.g.,  HWIL  facilities  or  installed  systems  test  facilities. 

The  SIT  results  suggest  strongly  that  ADS  has  good  potential  for  improving  PGM  testing.  The 
implication  is  that  test  planners  should  consider  the  technology  as  a  relevant  tool  for  their 
program  until  an  objective  assessment  suggests  otherwise. 

13.1.2  Command,  Control,  Communications,  Computer,  Intelligence,  Surveillance,  and 
Reconnaissance  Enhanced  Test  Capability 

The  ETE  Test  evaluated  the  utility  of  ADS  to  support  testing  of  C4ISR  systems.  The  test  used 
the  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  as  one  component  of  a 
representative  C4ISR  system.  The  ETE  Test  also  evaluated  the  capability  of  the  JADS  Test 
Control  and  Analysis  Center  (TCAC)  to  control  a  distributed  test  of  this  type  and  to  remotely 
monitor  and  analyze  test  results. 

The  ETE  Test  consisted  of  four  phases.  Phase  1  developed  or  modified  the  components  needed 
to  develop  the  ADS  test  environment.  Phase  2  used  the  ADS  test  environment  to  evaluate  the 
utility  of  ADS  to  support  DT&E  and  early  OT&E  of  a  C4ISR  system  in  a  laboratory  environment. 
Phase  3  transitioned  portions  of  the  architecture  to  the  E-8C  aircraft,  ensured  that  the  components 
functioned  properly,  and  checked  that  the  synthetic  environment  correctly  interacted  with  the 
aircraft  and  actual  light  ground  station  module  (LGSM).  Phase  4  evaluated  the  ability  to  perform 
test  and  evaluation  of  the  E-8C  and  LGSM  in  a  synthetically  enhanced  live  test  environment. 

The  test  concept  was  to  use  ADS  to  supplement  the  operational  environment  experienced  by  the 
E-8C  and  LGSM  operators.  By  mixing  available  live  targets  with  targets  generated  by  a 
constructive  model,  a  battle  array  approximating  the  major  systems  present  in  a  notional  corps 
area  of  interest  was  presented.  By  constructing  a  network  with  nodes  representing  appropriate 
command,  control,  communications,  computers  and  intelligence  (C4I)  and  weapon  systems,  a 
more  robust  cross  section  of  players  was  available  for  interaction  with  the  E-8C  and  LGSM 
operators. 

The  TCAC,  in  Albuquerque,  New  Mexico,  provided  test  control. 

The  ETE  Test  used  the  Janus  6.88D  simulation  to  generate  the  entities  representing  the  elements 
in  the  rear  of  a  threat  force.  The  U.S.  Army  Training  and  Doctrine  Command  Analysis  Center 
(TRAC)  at  White  Sands  Missile  Range  (WSMR),  New  Mexico,  provided  the  Janus  scenario  feed. 
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Bravo  Company,  303d  Military  Intelligence  Battalion  represented  the  LGSM  and  target  analysis 
cell  (TAC).  Fire  support,  in  the  form  of  the  Advanced  Field  Artillery  Tactical  Data  System 
(AFATDS),  was  represented  by  soldiers  from  the  4th  Infantry  Division  (Mechanized). 

Communications  among  these  systems  employed  such  doctrinally  correct  means  as  the  CGS-100, 
a  subsystem  of  the  Compartmented  All  Source  Analysis  System  (ASAS)  Message  Processing 
System  (CAMPS),  remote  workstations  (RWSs),  and  AFATDS  message  traffic. 

The  Tactical  Army  Fire  Support  Model  (TAFSM)  simulation  modeled  the  Army  Tactical  Missile 
System  (ATACMS)  battalion  and  sent  the  fire  and  detonate  protocol  data  units  (PDUs)  to  the 
Janus  6.88D  simulation.  Janus  then  modeled  the  engagement  results  and  reflected  them  in  the 
synthetic  environment. 

The  Joint  STARS  E-8C  simulation,  called  the  Virtual  Surveillance  Target  Attack  Radar  System 
(VSTARS),  represented  the  radar  subsystem  of  the  Joint  STARS  E-8C  in  a  laboratory 
environment.  It  was  composed  of  a  distributed  interactive  simulation  (DIS)  network  interface 
unit  (NIU),  a  radar  processor  simulator  and  integrator  (RPSI)  that  contained  the  two  real-time 
radar  simulations  with  necessary  databases,  and  various  simulations  of  E-8C  processes.  VSTARS 
was  operated  at  the  Northrop  Grumman  Surveillance  and  Battle  Management  Systems  facility  in 
Melbourne,  Florida. 

1 . 3 . 1 .2 . 1  End-to-End  Test  Results  and  Conclusions 

Within  the  narrow  confines  of  the  ETE  Test  data,  our  assessment  is  that  the  architecture  we 
employed  has  utility  for  support  of  T&E.  The  JADS  data  indicate  that  DT&E  and  OT&E 
activities  incorporating  ADS  technology  are  both  practical  and  cost  effective.  Our  broad 
conclusions  and  lessons  learned  can  be  summarized  as  follows. 

-  An  ADS  environment  can  enhance  C4ISR  system  DT&E  and  OT&E.  In  comparison  with 
conventional  tests,  ADS  allows  testers  to  examine  C4ISR  systems  under  realistic  conditions 
for  longer  periods  of  time,  over  far  larger  battlespaces,  and  at  a  much  lower  cost.  This 
versatile  technology  can  provide  test  environments  that  include  large  numbers  of  entities, 
entities  operating  under  realistic  but  unsafe  conditions,  and  joint  and  combined  operations. 
ADS  provides  C4ISR  system  testers  with  greater  flexibility  in  designing,  executing,  and 
analyzing  their  tests.  During  DT&E,  ADS  allows  for  more  realistic  compliance  testing  of 
C4ISR  subsystems  and  efficient  implementation  of  the  test-fix-verify  cycle  for  software 
development. 

-  ADS  testing  provides  dramatic  abilities  to  test  C4ISR  systems  or  components  in  that  system. 
For  instance,  in  the  ETE  Test  ADS  environment,  a  developmental  test  could  be  performed  on 
the  Joint  STARS  operations  and  control  subsystem  using  VSTARS  as  a  stimulus.  With  a 
similar  configuration,  an  operational  test  could  be  accomplished  on  a  LGSM.  The  ETE  Test 
ADS  environment  also  provides  ample  opportunities  to  install  new  components  for  various 
types  of  testing.  Links  to  airborne  weapon  system  simulators,  complimentary  sensor  feeds  or 
other  command  and  control  structures  can  be  easily  accomplished.  The  development  of  an 
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ADS  test  environment  during  system  development  greatly  improves  opportunities  for  C4ISR 
system  training  after  the  completion  of  the  test.  The  same  infrastructure  developed  for  testing 
can  be  modified  and  transitioned  to  a  training  environment  resulting  in  program  savings.  This 
technology  allows  C4ISR  system  operators  to  confirm  current  tactics,  try  “what-if  ’  scenarios 
and  new  tactics,  test  the  interoperability  and  compatibility  of  their  equipment,  and  gain  useful 
experience  in  a  realistic  operating  environment  containing  multiple  assets. 

-  The  ETE  Test  required  only  a  small  part  of  the  available  bandwidth  and  exhibited  a  low  PDU 
latency  rate.  The  ETE  Test  network  was  highly  reliable  during  testing,  due  largely  to  the  ETE 
Test  team’s  extensive  pretest  risk  reduction  efforts. 

-  Compared  to  conventional  testing,  ADS  testing  reduces  the  need  for  large  numbers  of  fielded 
personnel  and  vehicles.  The  ability  to  automatically  collect  and  analyze  test  data  also  reduces 
the  number  of  people  required  for  setup,  execution,  and  analysis.  ADS  test  success  relies  on 
well-organized  test  control  and  data  management  procedures.  Distributed  testing  requires 
sophisticated  instrumentation,  trained  personnel  to  operate  and  maintain  that  equipment,  and 
funds  to  support  personnel  and  equipment  at  distant  test  nodes. 

-  Testers  should  carefully  plan  the  development  of  the  simulations  and  links  comprising  their 
ADS  environment.  During  test  execution,  they  must  ensure  that  the  time  sources  are 
synchronized  and  continuously  monitor  PDU  traffic.  The  distributed  nature  of  ADS  testing 
necessitates  special  equipment  for  network  checkout  and  verification  and  requires  strict 
configuration  control  of  analysis  tools  and  collected  data. 

-  ADS  test  planning  should  be  detailed  enough  to  encompass  key  requirements  at  the  earliest 
possible  stages,  yet  flexible  enough  to  accommodate  unexpected  situations  during  test 
execution. 

-  A  conservative  development  approach  is  recommended.  Accomplish  risk  reduction  activities 
before  each  ADS  test  and  let  each  ADS  test  build  on  the  success  of  earlier  experiments. 
Successful  test  execution  requires  effective  intemode  communication,  test  and  resource 
control,  and  data  management  procedures. 

1.3.1 .2.2  Observations  for  C4ISR  T&E 

Through  a  process  of  inductive  reasoning,  we  can  transfer  some  of  the  specifics  of  the  ETE  Test 
to  the  general  class  of  C4ISR  systems.  In  the  general  case,  the  elements  of  the  ETE  Test 
architecture  are  basic  to  all  mission-level  testing  of  C4ISR  systems.  In  other  words,  there  is  a 
shooter,  a  sensor,  an  intended  target,  an  operating  environment,  and  a  test  control  center. 

The  shooter,  sensor,  and  target  can  be  represented  in  any  of  the  three  forms  associated  with 
distributed  simulation:  live,  virtual,  or  constructive.  We  do  not  see  a  one-for-one  transfer  of  ETE 
Test  techniques  to  other  tests.  Each  test  has  specific  requirements,  often  peculiar  to  the  particular 
system  under  test.  However,  we  do  see  a  transfer  of  the  principles,  design  processes,  and 
methodologies  used  in  the  ETE  Test. 
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Technically,  it’s  not  impossible  to  develop  a  high-fidelity  operating  environment  in  ADS. 
However,  the  costs  and  challenges  will  vary  from  test  to  test  and  will  depend  on  the  desired  level 
of  complexity. 

A  test  control  center  is  a  requirement  for  all  testing,  distributed  or  not.  Fortunately,  the  ETE  Test 
experience  suggests  that  the  control  center  can  function  from  almost  anywhere.  Thus,  a  specific 
test  may  save  resources  by  using  an  existing  control  center. 

The  ETE  Test  results  strongly  suggest  that  ADS  has  excellent  potential  for  improving  C4ISR 
testing.  Thus,  test  planners  should  consider  the  technology  as  a  relevant  tool  for  their  program 
until  an  objective  assessment  suggests  otherwise. 

1.3. 1.3  Electronic  Warfare  Enhanced  Test  Capability 

The  EW  Test  evaluated  the  utility  of  ADS  to  support  EW  T&E.  While  the  test  used  several 
efforts  to  examine  ADS-based  T&E,  the  cornerstone  effort  was  a  series  of  traditional  and  ADS- 
based  test  events  using  an  airborne  self-protection  jammer.  This  effort  was  called  the  self¬ 
protection  jammer  (SPJ)  test.  The  SPJ  test  defined  a  simple,  repeatable  test  scenario.  The 
scenario  was  executed  in  three  traditional  test  environments  to  create  a  data  baseline.  The  test 
scenario  was  then  executed  in  two  ADS-enhanced  test  environments.  The  first  ADS-based  test 
event  used  a  real-time  DSM  interacting  with  manned  threat  simulators  at  the  Air  Force  Electronic 
Warfare  Environment  Simulator  (AFEWES)  facility.  The  second  ADS-based  test  used  the  SPJ 
installed  on  an  F- 16  suspended  in  the  anechoic  chamber  at  the  Navy’s  Air  Combat  Environment 
Test  and  Evaluation  Facility  (ACETEF).  The  data  from  all  tests  were  statistically  compared  in  an 
attempt  to  quantify  the  impacts  of  ADS. 

1 . 3 . 1 .3 . 1  EW  Test  Results  and  Conclusions 

Within  the  confines  of  the  SPJ  test  data,  JADS  concluded  that  ADS  architectures  that  allow  the 
capabilities  of  geographically  separated  facilities  to  be  combined  to  create  a  realistic  test 
environment  for  EW  devices  can  be  designed.  This  allows  the  same  test  environment  to  be  used 
for  SUT  representations  ranging  from  DSMs  to  operational  equipment.  Testing  in  a  common 
ADS-based  environment  represents  a  significant  departure  from  the  traditional  sequential  EW  test 
process.  EW  Test  key  results  are  identified. 

-  Designing  ADS  architectures  requires  a  close  team  comprised  of  several  technical  experts 
spanning  several  disciplines  directed  by  a  system  integrator. 

-  The  architecture  produced  valid  results  for  both  the  DSM  and  actual  jammer  hardware. 

-  Latency  within  the  closed-loop  interaction  was  aggressively  managed,  and  JADS  was  able  to 
meet  its  objective  for  more  than  95%  of  the  runs. 

-  The  high  level  architecture  (HLA)  appears  to  be  a  feasible  method  for  linking  simulations  for 
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T&E.  It  is  appropriate  to  use  HLA  to  link  to  other  HLA-compliant  simulations/simulators, 
but  the  T&E  community  should  not  view  it  as  the  only  architecture  to  consider  when 
designing  distributed  tests. 

-  Two  of  the  eleven  EW  test  facilities  surveyed  in  1996  as  part  of  the  Threat  Simulator  Linking 
Activity  (TSLA)  effort  that  were  appropriate  for  ADS-based  testing  have  been  closed.  While 
this  is  a  significant  erosion  in  the  infrastructure  needed  to  design  and  execute  ADS-based 
tests,  it  already  limits  the  traditional  EW  testing  process. 

1 .3 . 1 .3 .2  Observations  for  EW  T&E 

JADS  assessment,  based  on  the  different  EW  Test  efforts,  is  that  ADS  has  varying  levels  of  utility 
for  EW  T&E.  These  levels  of  utility  depend  on  the  nature  of  the  EW  device  being  tested  and  the 
availability  of  suitable  test  facilities.  Single  function  EW  devices  and  federated  EW  systems  are 
expected  to  benefit  least  from  an  ADS-enhanced  test  process.  Only  radio  frequency  (RF) 
jammers  may  see  sufficient  benefit  to  outweigh  the  additional  cost  of  an  ADS-enhanced  test 
process.  Integrated  EW  systems  may  see  significant  benefits  where  a  single  test  facility  is  not 
capable  of  providing  all  the  stimulation  (including  the  closed-loop  SUT  versus  manned  threat 
interaction  for  systems  that  include  RF  jammers)  needed  to  simultaneously  test  all  the  particular 
integrated  EW  system’s  functions.  Systems  of  systems  testing  such  as  that  required  for  electronic 
support  (ES)  systems  should  see  significant  benefits  in  ADS-based  testing.  By  breaking  through 
traditional  facility  barriers  and  allowing  multiple  facilities  to  participate  in  the  same  test  event,  the 
following  types  of  test  events  are  possible. 

Combined  DSM/  hardware-in-the-loop  (HITL)  events.  JADS  demonstrated  this  in  Phase  2  of 
the  SPJ  test.  This  combination  allowed  a  real-time  DSM  to  interact  with  manned  threats  before 
any  actual  SUT  hardware  was  developed.  Problems  found  here  are  quite  inexpensive  to  fix. 
Additionally,  this  may  provide  a  way  to  gain  confidence  that  the  DSM  is  behaving  as  the  real 
hardware  which  in  turn  may  increase  confidence  in  model-on-model  excursions  to  expand  the  test 
envelope.  This  also  allows  the  possibility  to  directly  compare  dissimilar  technology  during  an 
analysis  of  alternatives.  Here  the  limitation  is  that  the  model  must  run  in  real  time.  This  is  more 
strict  than  compliance  with  either  the  Joint  Modeling  And  Simulation  System  (JMASS)  or  the 
HLA. 

Combined  system  integration  laboratory  (SIL)/HITL  events.  Breadboard  and  brassboard 
hardware  can  be  transported  to  the  HITL  and  tested  today.  Leaving  the  SUT  in  its  development 
site  may  reduce  the  time  required  to  solve  problems  that  are  encountered  and  also  allows  the 
other  offensive  or  defensive  systems  and  operators  to  play  in  an  environment  with  manned  threats. 
This  means  more  realistic  and  reactive  testing  early.  Here  the  limitation  is  the  capability  of  the 
stimulators  to  recreate  the  environment. 
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Combined  HITL/installed  systems  test  facility  (ISTF)  events.  This  expands  tests  where  the 
SUT  is  integrated  with  the  host  platform.  JADS  demonstrated  this  in  the  Phase  3  test. 
Traditionally  this  is  where  electromagnetic  interference  (EMI)/compatibility  (EMC)  testing  occurs 
and  stimulation  is  limited  to  what  the  ISTF  can  provide.  Manned  threats  are  not  usually  located  in 
ISTFs.  The  HITL/ISTF  combination  allows  closed-loop  effectiveness  testing  to  occur  at  the 
same  time  as  EMI/EMC  testing.  This  should  avoid  sequential  testing  should  fixes  to  EMI/EMC 
testing  impact  the  SUT. 

Combined  open  air  range  (OAR)/HITL  events.  This  expands  or  improves  the  threat  set  that 
can  be  applied.  Manned  semi-active  threats  can  be  combined  with  remote  seekers  to  improve  the 
missile  flyout  simulation.  Unmanned  training  emitters  can  be  used  with  HITL  operators  to  add 
threats.  In  this  case  the  live  platform  position  can  be  sent  to  the  HITL  allowing  those  threat 
operators  to  engage  a  synthetic  target.  The  threat  modes  and  actions  are  sent  back  to  the  OAR 
and  recreated  by  the  training  emitter. 

Connecting  facilities  derives  additional  benefits.  Both  integrated  systems  and  systems  of  systems 
are  expected  to  benefit  from  distributed  testing. 

Adequate  testing  of  integrated  systems  requires  a  simultaneous  test  of  all  integrated  EW  functions 
to  include  closed-loop  SUT/manned  threat  interactions  in  a  common  environment  in  a  single  test 
event.  It  is  possible  to  separate  the  functions  and  test  each  sequentially.  However,  this  adds  risk 
to  the  acquisition  decision.  Where  a  single  facility  is  capable  of  testing  an  integrated  EW  system 
to  this  level,  ADS  is  expected  to  provide  limited  benefit.  ADS  offers  the  opportunity  to  use 
existing  facilities  instead  of  expanding  a  single  facility  to  provide  an  adequate  level  of  test 
capability. 

EW  systems  are  not  limited  to  single  platform  scenarios.  The  class  of  EW  systems  called 
electronic  support  deals  with  system  of  systems  EW  actions.  Testing  a  system  of  systems 
traditionally  occurs  either  in  model-on-model  or  OAR  events.  Both  are  limited  and  OAR  events 
can  be  quite  expensive.  More  importantly,  by  creating  a  reactive,  operationally  realistic 
environment  through  ADS,  the  contribution  of  EW  systems  may  be  captured  directly  in  mission- 
level  measures  of  effectiveness. 

1.3.2  Cost  Reducing  Potential 

Another  way  of  categorizing  the  benefits  relates  to  the  cost  reducing  potential  of  distributed 
testing.  The  use  of  distributed  testing  may  directly  result  in  cost  savings  and/or  reduce  overall 
program  costs  by  avoiding  costs. 

1. 3.2.1  Cost  Savings 

Live  tests  requiring  that  multiple  platforms  be  brought  to  and  assembled  at  a  test  range  is  one 
example  of  the  type  of  test  that,  for  some  cases,  would  be  more  efficiently  and  effectively 
accomplished  through  distributed  testing  or  would  at  least  support  reduction  in  the  number  of 
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expensive  live  tests  required.  Software  (SW)  developed  for  distributed  testing  may  be  reused  to 
support  traditional  testing  methods.  One  example  of  this  is  from  the  JADS  SIT  LFP.  The  special 
purpose  interface  developed  to  connect  aircraft  to  the  Advanced  Medium  Range  Air-to-Air 
Missile  (AMRAAM)  ground  lab  was  later  used  for  troubleshooting  in  traditional  testing  methods. 
The  VSTARS  is  probably  the  best  example  of  SW  developed  for  distributed  testing  with  reuse 
capability.  However,  more  than  software  fits  in  this  category.  For  example,  the  AFEWES 
manned  simulators  were  used  for  decades  in  traditional  T&E.  They  were  used  by  JADS  for  ADS- 
based  testing  by  adding  an  interface  to  the  facility.  Any  potential  overlap  between  traditional  and 
distributed  testing  methods  should  be  addressed  by  the  feasibility  analysis. 

The  empirical  data  from  the  JADS  tests  have  provided  the  JTF,  already  experienced  in 
conventional  T&E  techniques,  with  sufficient  confidence  to  identify  benefits  and  savings  arising 
from  the  use  of  distributed  testing.  Several  uses  for  distributed  testing  technologies  have  been 
identified  that  support  cost  savings  for  traditional  tests  and  across  the  program  that  should  be 
considered  when  assessing  the  cost  of  distributed  testing  to  the  overall  program. 

-  Distributed  testing  analysis  results  can  indicate  where  live  testing  can  best  be  focused  and  may 
reduce  the  need  for  some  live  tests. 

-  Distributed  testing  results  in  a  synthetic  environment  (SE)  that  can  support  other  areas  of  the 
program,  other  programs,  and  other  Department  of  Defense  (DoD)  initiatives.  For  example: 

-  The  SE  capability  supports  system  design  and  development,  training  simulation,  and 
training  for  the  live  test  community. 

-  It  can  be  used  for  early  operational  assessments,  development  of  tactics,  techniques  and 
procedures  before  system  testing,  test  rehearsal,  verification  of  data  sources  and  data 
reduction  techniques,  and  to  determine  whether  adequate  data  are  collected  to  evaluate 
test  measures. 

-  In  some  cases,  other  systems’  test  programs  that  require  similar  input  can  reuse  an  SE 
(e.g.,  the  ETE  Test  SE,  with  minor  upgrades,  could  be  reused  for  the  Block  2  ATACMS 
OT&E. 

The  Simulation  Based  Acquisition  (SBA)  and  Simulation  T&E  Process  (STEP)  initiatives  will 
advance  more  quickly  as  programs  initiate  development  of  SEs  as  a  normal  step  in  the  program’s 
life  cycle.  The  focus  of  SBA  is  reduced  cycle  time  -  the  ability  to  use  the  technology  to  develop 
systems  more  rapidly,  reducing  the  current  major  system  cycle  time  from  15-20  years  to  7-10 
years  (50  percent).  Currently,  most  systems  progress  through  a  series  of  sequential  tests  at 
multiple  facilities  using  the  unique  capabilities  of  the  individual  facilities  to  address  different  test 
objectives.  Distributed  testing  technologies  can  be  used  to  link  the  individual  facilities  and 
conduct  concurrent  testing  of  multiple  test  objectives,  thus  providing  the  opportunity  for 
significant  time  and  cost  savings.  If  shortening  the  acquisition  timeline  results  in  cost  savings, 
then  there  is  a  powerful  argument  for  using  the  technology.  Major  test  programs  often  employ  a 
variety  of  tools  including  physical  models,  force-on-force  models,  HITL  facilities,  open  air  testing, 
SIL,  simulators,  measurement  facilities  and  the  like.  In  some  cases,  testers  take  advantage  of 
large-scale  field  exercises  (which  are  becoming  increasingly  rare).  In  many  cases,  the  SUT  must 
be  moved  from  facility  to  facility  in  a  sequential  stream.  Because  of  scheduling  issues  with  high 
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use  facilities,  there  is  often  a  significant  amount  of  slack  in  test  program  time  lines.  Where  large- 
scale  field  exercises  are  a  test  vehicle  of  choice,  there  may  be  a  year  or  more  between  test 
opportunities.  Additionally,  the  use  of  distributed  testing  can  reduce  total  testing  costs  by 
replacing  a  limited  number  of  live  tests  with  distributed  tests. 

1. 3.2.2  Cost  Avoidance 

Cost  avoidance  is  the  notion  that  distributed  testing  can  help  perform  more  complete  testing 
earlier  in  a  program  and  identify  failure  modes  and  other  problems  earlier  when  they  can  be  fixed 
cheaper  and  in  less  time  than  when  they  are  discovered  later  in  the  system’s  life  cycle.  For 
example,  the  ability  of  distributed  testing  to  incorporate  man-in-the-loop  provides  an  opportunity 
for  cost  avoidance  when  the  technology  is  used  as  a  test  rehearsal  and  training  tool.  Test 
participants  can  climb  the  early  part  of  the  learning  curve  in  the  virtual  environment,  and  the  test 
schemes  can  be  refined  prior  to  using  very  expensive  test  facilities.  The  time  associated  with  lost 
test  events  could  therefore  be  reduced.  In  some  situations  distributed  testing  can  reduce  the  risk 
and  cost  of  wasted  live  fire  attempts  by  providing  a  realistic  test  rehearsal  capability.  More 
thorough  testing  should  result  in  identification  of  system  deficiencies  earlier  in  the  life  cycle  when 
they  can  be  fixed  more  efficiently,  thus  saving  schedule  and  dollars.  Distributed  testing  can  redo 
live  tests  that  have  encountered  problems  more  quickly  than  live  retesting  which  would  result  in 
schedule  slips.  Traditional  tests  are  still  required,  but  by  reducing  the  number  of  lost  test  events, 
you  can  save  money  over  all. 

Testing  could  be  more  thorough,  could  complete  more  test  scenarios,  and  could  be  used  for  cases 
where  live  tests  are  limited  by  test  range  constraints,  weather,  equipment,  or  when  the  test 
environment  needs  to  be  very  complex.  For  example,  the  Joint  STARS  Multiservice  Operational 
Test  and  Evaluation  (MOT&E)  was  originally  planned  to  be  conducted  over  the  National  Training 
Center  at  Fort  Irwin,  California,  with  300-500  military  vehicles  in  the  maneuver  area. 
Background  traffic  from  Los  Angeles,  California,  and  Interstate  10  would  have  provided  a  more 
operational  load  on  the  system,  however,  would  not  have  provided  an  increased  load  on  the 
operators.  This  approach  was  taken  because  the  test  planners  realized  they  would  never  be  able 
to  provide  an  operationally  representative  test  environment  short  of  an  actual  war.  The  JADS 
ETE  Test  was  able  to  represent  10,000  vehicles  arranged  and  maneuvering  in  a  doctrinally  correct 
threat  laydown  of  enemy  corps.  This  provided  a  much  more  representative  operational 
environment  to  perform  operational  testing  and  provided  the  testers  with  a  repeatable 
environment  where  ground  truth  was  known. 

1.4  Deciding  Whether  to  Use  Distributed  Testing  Technology 

In  deciding  whether  to  use  distributed  testing,  the  tester  should  first  determine  its  cost 
effectiveness.  To  do  this  the  tester  would  first  estimate  the  potential  benefits  of  implementing 
distributed  testing  and  determine  any  testing  constraints  that  might  limit  its  implementation.  To 
help  determine  the  feasibility  and  utility  of  distributed  testing  test  methods,  JADS  published  A 
Test  Planning  Methodology  -  From  Concept  Development  Through  Test  Execution  which  can  be 
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found  at  www.iads.abq.com.3  This  report  provides  a  well-defined  process  to  support  the 
feasibility  analysis  presented  in  the  proposed  WBS  element  1  .X.O.  This  should  cover  the  technical 
requirements  of  the  system’s  tests,  the  performance  metrics  needed,  risks  associated  with  not 
using  distributed  testing  test  methods  (e.g.,  insufficient  testing),  and  rough  order  of  magnitude 
(ROM)  costs  and  schedule  estimates. 

After  identifying  potential  benefits  and  constraints  the  PM  would  estimate  the  costs  with  and 
without  distributed  testing.  Some  cost  considerations  when  deciding  to  use  distributed  testing  are 
the  availability  of  a  DSM  for  the  project,  fidelity  and  complexity  of  the  technical  requirements, 
and  availability  and  maturity  of  other  federation  entities.  Balancing  the  benefits  and  costs  would 
allow  the  structure  of  an  optimal/near  optimal  distributed  testing  program.  Any  feasibility  analysis 
should  provide  information  on  when,  where,  and  how  distributed  testing  methods  can  best  be 
applied  to  complement  traditional  test  methods  and  enhance  the  overall  T&E  phase.  The  JADS 
experience  has  shown  that  distributed  testing  technologies  can  complement  other  T&E 
approaches  but  should  not  necessarily  replace  more  traditional  forms  of  testing  (e.g.,  live  testing). 

To  illustrate  the  decision  process,  this  report  outlines  three  possible  approaches  to  the  application 
of  distributed  testing:  (1)  a  cost  savings  approach,  (2)  a  cost  neutral  approach  and  (3)  a  more 
effective  testing  approach.  In  the  first  approach,  substituting  distributed  testing  for  selected  live 
testing  reduces  total  testing  costs.  In  the  second  case,  distributed  testing  is  added  and  the  scope 
of  other  testing  is  reduced.  The  third  approach  adds  distributed  testing  without  eliminating  any 
live  testing,  so  that  total  costs  are  higher,  but  with  the  benefit  of  improved  testing. 

In  the  case  of  reduced  testing  costs,  the  total  testing  budget  is  reduced  (although  the  total 
program  budget  would  undoubtedly  be  kept  fixed,  providing  more  “cushion”  to  the  overall 
program)  because  distributed  testing  allows  the  scope  of  the  expensive  live  testing  to  be 
significantly  reduced  and  because  the  distributed  testing  costs  are  less  than  the  live  tests  replaced. 
This  approach  would  be  very  case  specific  and  problematic  for  the  PM,  since  some  live  testing  is 
required  and  the  number  of  live  trials  is  usually  minimal  anyway. 

In  the  case  of  costs  remaining  the  same,  the  total  testing  budget  is  kept  fixed,  and  distributed 
testing  is  added  by  reducing  the  scope  of  testing  with  the  other  techniques  (i.e.,  the  zero  sum 
approach),  including  the  expensive  live  testing.  In  this  case,  distributed  testing  has  no  cost  impact 
or  savings  (cost  neutral),  since  the  total  testing  budget  is  unchanged.  The  justification  in  this  case 
would  be  that  distributed  testing  allows  you  to  do  better  tests  which  would  reduce  the  potential 
for  expensive  fixes  later  in  the  program.  The  last  scenario  would  apparently  involve  increased 
testing  costs,  but  if  waste  of  time  and  resources  is  avoided  costs  may  balance  out. 

In  the  case  of  increased  testing  costs,  distributed  testing  is  added  without  changing  the  scope  of 
the  other  testing  techniques.  Given  this  approach,  the  expectation  would  be  that  the  enhanced 


3  After  1  March  2001  refer  requests  to  Headquarters  Air  Force  Operational  Test  and  Evaluation 
Center  History  Office  (HQ  AFOTEC/HO),  8500  Gibson  Blvd.  SE,  Kirtland  Air  Force  Base,  New 
Mexico  87117-5558,  or  Science  Applications  International  Corporation  (SAIC)  Technical 
Library,  2001  North  Beauregard  St.,  Suite  80,  Alexandria,  Virginia  22311 . 
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testing  would  pay  for  itself  in  delivery  of  a  better  system  to  the  end  user,  or  that  system 
deficiencies  that  may  go  undetected  without  distributed  testing  could  be  addressed  earlier  with 
attendant  cost  savings. 
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2.0  Cost  Guidance 


Section  1  discussed  the  potential  to  either  save  on  costs  or  to  avoid  costs.  It  proposes  three  cases 
where  a  program  manager  may  need  to  estimate  costs  or  perform  cost  trade-offs  between  possible 
test  methodologies.  This  section  provides  cost  guidance  to  allow  the  program  manager  to  make 
estimates  and  perform  the  trade-offs.  Developed  by  MITRE  Corporation’s  EDAC  and  supported 
by  JADS,  this  section  provides  information  that  T&E  organizations  may  find  useful  in  costing 
testing  methods  and  processes. 

2.1  Purpose  and  Focus 

Initially,  JADS  considered  developing  a  parametric  model  to  estimate  the  costs  of  employing 
distributed  testing  in  the  DT&E  phase  of  system  acquisition  or  the  costs  of  using  distributed 
testing  for  OT&E.  JADS  was  to  develop  the  model  from  information  collected  during  the 
execution  of  the  three  JADS  tests.  As  it  turned  out,  however,  this  was  not  practical.  The  tests 
did  not  provide  enough  of  a  database  to  develop  a  parametric  model.  Specific  cost  information 
was  not  collected  to  the  level  necessary  for  the  development  of  a  parametric  cost  model,  largely 
because  of  the  lack  of  insight  into  the  internal  cost  accounting  of  the  various  test  organizations 
and  laboratories  JADS  worked  with.  Additionally,  JADS  cost  data  were  specific  only  to  our 
particular  tests.  Although  such  data  might  provide  some  indication  of  the  costs  of  using 
distributed  testing  in  a  similar  program,  they  do  not  necessarily  provide  information  relevant  to 
other  specific  system  types.  Therefore  the  cost  guidance  that  follows  is  based  on  the  JADS  test 
experience  and  is  presented  at  a  level  above  a  cost  model.  As  a  result,  it  is  simply  a  useful  first 
step  in  supporting  a  given  program’s  distributed  testing  cost  estimating  efforts.  For  example,  the 
cost  guidance  addresses  cost  drivers,  major  areas  of  risk,  risk  mitigation  and  lessons  learned. 
Although  the  cost  guidance  falls  short  in  providing  a  complete  estimate  of  distributed  testing  costs 
for  every  specific  requirement,  it  will  help  PMs  begin  to  determine  the  costs  of  incorporating 
distributed  testing  into  their  T&E  program. 

2.1.1  Cost  Guidance  Section  Contents 

This  cost  guidance  section  focuses  on  those  T&E  activities  where  distributed  testing  potentially 
has  an  impact.  Available  cost  data  were  reviewed  and  key  subject  matter  experts  were 
interviewed  to  determine  potential  cost  drivers,  lessons  learned,  risks,  and  risk  mitigation 
strategies.  It  includes  information  pertaining  to  the  identification  of  relevant  cost  elements  for 
using  distributed  testing  for  T&E.  This  section  also  provides  lessons  learned,  identifies  potential 
risk  areas,  and  identifies  tools  to  support  the  use  of  distributed  testing. 

This  section  includes  information  on  the  cost  guidance  purpose,  intended  use,  and  approach  used 
to  identify  the  guidance.  General  and  specific  findings  that  could  be  used  as  guidance  by  a  PM  in 
deciding  to  use  distributed  testing  for  the  T&E  phase  are  presented  in  Section  2.2.  Section  2.3 
provides  cost  guidance  identified  primarily,  but  not  only,  from  the  JADS  experience.  Section  2.4 
presents  the  proposed  WBS  with  definitions  of  each  element  and  Section  2.5  provides  information 
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on  tools  that  support  the  use  of  distributed  testing  in  the  T&E  phase  as  well  as  sources  for  other 
pertinent  information. 

2.1.2  Intended  Uses 

Originally,  JADS  requested  a  cost  model  to  support  the  development  of  incorporating  distributed 
testing  activities  into  a  project’s  T&E  phase.  After  further  investigation  we  concluded  that  this 
was  not  feasible  given  the  limited  amount  of  information  available  from  the  three  JADS  tests  and 
other  sources.  Another  factor  limiting  the  use  of  JADS  test  data  for  development  of  a  cost  model 
was  that  the  JADS  tests  were  optimized  for  experimental  objectives  and  not  for  evaluating  the 
SUT  objectives.  Data  from  these  experiments  would  not  necessarily  be  reflective  of  an  actual  test 
program’s  experience.  As  a  result,  JADS  test  data  could  not  be  used  to  support  development  of 
accurate  and  reasonable  cost  estimating  relationships  (CERs)  for  actual  projects. 

In  this  report,  JADS  provides  PMs  and  their  T&E  engineering  teams  with  information  on  what  it 
identified  as  cost  drivers,  risks,  risk  mitigating  strategies,  and  lessons  learned.  In  some  cases, 
costs  encountered  in  the  JADS  tests  are  included  for  the  purpose  of  providing  program  cost 
analysts  with  an  initial  data  point.  Certainly,  the  T&E  community  will  need  to  collect  much  more 
data  before  meaningful  CERs  or  cost  models  can  be  developed  that  reflect  distributed  testing  test 
costs. 

2.1.3  Approach 

The  approach  used  to  identify  relevant  cost  guidance  associated  with  the  use  of  distributed  testing 
in  the  T&E  phase  included  the  following  steps. 

-  Interview  JADS  members  involved  in  all  aspects  of  the  three  JADS  tests  to  understand  what  is 
required  to  incorporate  distributed  testing  into  the  T&E  process,  that  is,  the  differences  from 
conventional  T&E  approaches,  and  to  define  a  new  process  that  considers  and  possibly  uses 
distributed  testing  technology. 

-  Discuss  distributed  testing  applications  with  other  organizations  using  distributed  testing  or 
related  technologies  including 

-  U.S.  Army  Simulation,  Training,  and  Instrumentation  Command  (STRICOM)  Advanced 
Distributed  Simulation  Technology  II  (ADST-II)  Program  Office  website  at 
http://www.stricom.army.mil/PRODUCTS/ADSTII/; 

-  U.S.  Air  Force  Studies  and  Analysis  Agency  (AFSAA)  Theater  Battle  Arena  (TBA)  office 
website  at  http://www.afsaa.hq.af.mil/SAM/SAMT/;  and 

-  Defense  Modeling  and  Simulation  Office  (DMSO)  High  Level  Architecture  (HLA) 
Federation  and  Development  Process  (FEDEP)  developers  website  at 
http://hla.dmso.mil/hla/federation/fedep/. 

-  Note  that  while  distributed  testing  technologies  are  being  used  by  these  and  other 
organizations,  none  were  using  it  directly  in  the  T&E  phase  (except  the  FEDEP  developers 
who  supported  the  JADS  EW  Test).  None  of  these  organizations’  distributed  testing  use 
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fulfilled  the  more  stringent  T&E  requirements  for  latency,  fidelity,  instrumentation  and  data 
management.  The  FEDEP  checklist  and  development  of  an  HLA  federation  were  used  on  the 
JADSEWTest. 

-  Identify  elements  of  a  generic  T&E  portion  of  a  WBS  that  would  reflect  incorporation  of 
distributed  testing  in  that  phase.  This  proposed  WBS  captures  activities  either  new  to  the 
T&E  phase  or  that  will  require  modification  resulting  from  distributed  testing.  The  proposed 
WBS  was  reviewed  by  JADS  and  revised  based  on  their  inputs.  It  was  also  compared  against 
the  DMSO  FEDEP  Checklist  1,  and  DMSO  DoD  Verification,  Validation  and  Accreditation 
( W&A )  Recommended  Practices  Guide,  November  1996. 

-  We  have  not  attempted  to  modify  WBS  elements  outside  of  the  system  T&E  element,  but  if 
distributed  testing  is  used  for  this  phase,  the  PM  office  (PMO)  should  identify  other  areas  in 
the  program  that  would  be  impacted.  For  example,  systems  engineering  and  program 
management  (SEPM)  may  need  additional  resources,  while  traditional  T&E  activities  may 
realize  savings. 

-  Gather  and  analyze  JADS  experience  and  map  that  experience  to  specific  elements  in  the 
proposed  WBS. 

-  Identify  resources  for  PMs  to  access  to  support  distributed  testing  use  in  T&E:  tools,  HLA 
information,  upgraded  test  ranges,  and  virtual  environments. 

-  Make  recommendations  for  changes  within  the  DoD  community  to  accommodate  future 
applications  of  distributed  testing  technology  to  T&E  requirements. 

2.2  Findings 

Findings  from  the  cost  guidance  study  are  presented  in  two  categories.  The  first  category  consists 
of  general  findings  on  distributed  testing  technology,  which  could  be  used  in  other  phases  of  the 
program  and  should  be  considered  during  the  feasibility  analysis.  The  second  category  consists  of 
specific  findings,  which  describe  characteristics  specific  to  the  T&E  phase,  but  that  are  not 
specific  to  one  WBS  element. 

2.2.1  General  Findings 

-  Since  JADS  was  chartered  in  1994,  enabling  technologies  associated  with  distributed  testing 
have  made  significant  progress.  One  example  is  the  HLA  protocol.  HLA  software,  training, 
and  use  have  increased  dramatically.  Certification  of  HLA  federates  has  increased  and  test 
ranges  are  updating  to  accommodate  these  new  technologies. 

-  Technology  advancements  should  significantly  reduce  some  of  the  difficulties  reported  by 
JADS.  Distributed  testing,  HLA,  and  other  technologies  and  protocols  are  making  integrated 
environments  much  more  feasible  and  available  to  the  program  developer.  Integrated 
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synthetic  environments  allow  programs  to  do  more  in  less  time  with  better  results  than 
existing  approaches  used  today. 

-  Distributed  testing  technologies  support  three  major  DoD  initiatives. 

-  DoD  acquisition  reform,  DoD  Regulation  5000.2,  Mandatory  Procedures  for  Major 
Defense  Acquisition  Programs  (MDAPS)  and  Major  Automated  Information  System 
(MAIS)  Acquisition  Programs,  requires  modeling  and  simulation  to  be  an  integral  part  of 
test  planning  and  that  T&E  planning  start  in  Phase  0; 

-  Simulation  Based  Acquisition  (SBA),  as  defined  in  the  SBA  Roadmap  produced  by  the 
Joint  SBA  Task  Force  which  was  chartered  by  the  DoD  Executive  Council  for  Modeling 
and  Simulation  (EXCIMS);  and 

-  Simulation  T&E  Process  (STEP)  initiated  by  the  Office  of  the  Under  Secretary  for 
Defense,  Acquisition  and  Technology  (OUSD(A&T))  in  support  of  acquisition 
streamlining. 

These  initiatives  are  complimented  by,  and  provide  support  to  distributed  testing  use.  As  these 
initiatives  are  implemented,  models  and  simulations  will  be  developed  for  the  entire  program 
lifecycle  and  significant  costs  will  not  have  to  be  borne  by  the  T&E  phase  only. 

-  The  DoD  life  cycle  cost  model  needs  to  be  modified  to  support  T&E  activities  at  the 
beginning  of  the  program.  The  sooner  distributed  testing  is  started,  the  sooner  design  flaws 
and  bugs  will  be  found  and  changed.  As  a  general  rule,  the  sooner  you  find  a  problem,  the 
lower  the  cost  to  fix  the  problem. 

-  Staff  trained  in  new  technical  disciplines  will  be  required  to  successfully  implement  distributed 
testing  technology.  For  example,  network  engineering  is  critical  to  the  successful  application 
of  distributed  testing  since  the  instrumentation  must  be  able  to  account  for  network 
availability,  bandwidth  usage  and  network  induced  latency.  Another  example  is  the 
coordination  of  test  control  and  analysis  communications,  visual  displays,  and  computer 
processing  capability  to  ensure  integrity  of  responses,  a  key  factor  in  T&E  exercises. 

2.2.2  Specific  Findings 

Implementing  the  three  JADS  tests  provided  many  lessons  learned  that  should  be  considered  and 

incorporated  into  the  PMO’s  cost  estimate  addressing  distributed  testing. 

-  In  almost  all  interviews  with  JADS  personnel,  strong  systems  engineering  processes  including 
configuration  management  were  identified  as  key  to  successful  distributed  testing,  because  this 
testing  approach  requires  significant  coordination  among  many  entities  within  a  testing 
federation. 

-  Pro-active  program  management  and  leadership  will  encourage  team  communication,  and 
clear  delegation  of  authority  will  keep  communication  lines  open. 
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-  Intensive  planning  is  needed  to  ensure  all  federates  are  on  schedule  for  development  of  or 
modifications  to  player  representations,  test  rehearsals  and  execution,  synchronization,  and 
interoperability.  Contingency  planning  will  reduce  risks. 

-  Documented  requirements  and  acceptance  criteria  will  support  efficiency  and  cost 
containment  for  W&A  activities.  W&A  planning  should  be  part  of  the  feasibility  analysis 
and  Phase  0  activities.  Note,  that  the  verification  and  validation  (V&V)  of  models  and 
distributed  simulations  are  formally  required  by  DoD  Instruction  5000.61,  DoD  Modeling  and 
Simulation  (M&S)  Verification,  Validation  and  Accreditation  (W&A),  before  they  can  be 
used  in  testing.  Simulations  have  to  go  through  a  V&V  process  at  the  federate  level  and  at 
the  federation  level. 

-  Effective  data  management  is  essential  since  distributed  testing  produces  copious  amounts  of 
data.  A  comprehensive  data  management  plan  must  include  explicit  information  on  the  data  to 
be  collected  at  each  network  node,  on-site  data  processing  requirements,  and  data  that  will  be 
transmitted  to  the  analysis  center. 

2.3  Cost  Guidance 

This  section  gives  guidance  on  factors  that  should  be  considered  in  building  a  cost  estimate  for 
distributed  testing  and  that  may  drive  testing  costs  beyond  available  resources.  This  information 
is  provided  in  three  sections:  general  cost  drivers,  risk  factors  that  affect  cost  drivers,  and 
guidance  for  specific  cost  elements  in  the  proposed  WBS.  (The  proposed  WBS  with  definitions  is 
in  Section  2.5.) 

2.3.1  General  Cost  Drivers 

Several  cost  drivers  were  identified  during  the  execution  of  the  three  JADS  tests  that  have  impact 
on  many  cost  elements  in  the  proposed  WBS.  They  are  listed  here  to  emphasize  the  importance 
of  preparing  and  planning  for  their  minimization. 

-  SE  complexity,  as  defined  by  the  number  and  types  of  nodes,  interfaces,  and  bandwidth 
requirements,  will  increase  costs  and  risks  to  all  phases  of  the  distributed  portion  of  the  test 
program.  Since  a  major  virtue  of  distributed  testing  is  to  support  the  construction  of  complex 
test  environments,  the  test  planner  must  balance  the  need  for  complexity  against  the  expense 
and  fragility  of  the  test  architecture. 

-  Fidelity  requirements  for  the  models  can  range  from  simple,  script-driven  models  to  high- 
fidelity  man-in-the-loop  simulators.  Fidelity  and  costs  are  directly  proportional. 

-  The  JADS  experience  was  that  test  ranges  and  labs  underestimated  costs  10  to  15  percent.  It 
is  recommended  that  agreements  with  these  organizations  be  formalized  in  a  statement  of 
capabilities  and  that  they  be  required  to  provide  detailed  cost  estimates.  The  program  test’s 
priority  number  should  be  provided  to  the  PMO’s  T&E  representative.  Additionally,  any 
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provisions  for  schedule-delay  penalties  should  be  agreed  to.  (These  penalties  are  one  of  the 
costs  of  poor  scheduling.) 

-  Test  ranges,  labs,  and  other  federates  should  have  experienced  staff  trained  in  distributed 
testing  and  HLA  application.  If  they  do  not  have  this  capability,  increase  the  cost  estimate  for 
elements  associated  with  their  support. 

-  JADS  found  that  configuration  management  of  hardware  (HW),  software  (SW),  network 
interface  units  (NIU),  and  models  was  more  important  to  distributed  testing  than  to  traditional 
testing.  A  detailed  configuration  management  plan  and  implementation  process  should 
minimize  this  risk. 

-  The  cost  of  implementing  distributed  testing  can  be  significantly  reduced  if  distributed  testing 
is  planned  for  within  the  larger  test  program,  i.e.,  while  traditional  testing  approaches  are 
being  developed. 

-  Architecture  integration  should  be  fully  understood  to  properly  estimate  it  at  the  outset  of  the 
program.  JADS  found  this  activity  to  be  consistently  underestimated. 

-  Legacy  test  ranges  with  out-of-date  equipment  caused  significant  problems  for  the  JADS 
tests.  This  could  also  be  clarified  through  a  formal  document  from  the  test  range. 

-  JADS  recommends  that  verification  be  done  on  network  nodes  as  they  are  developed. 

2.3.2  Risk  Factors  Impacting  Costs 

-  Time  synchronization,  instrumentation  and  data  management  are  all  critical  to  the  success  of 
distributed  testing.  Planning  for  their  successful  implementation  will  minimize  the  risk  of  cost 
increases. 

-  Because  distributed  testing  required  multiple  federates  to  set  up  the  SE,  scheduling  and 
coordination  became  much  more  complex  and  difficult.  This  characteristic  affects  all  phases 
of  the  test  program.  Careful  planning  will  mitigate  some  of  the  risk. 

-  As  in  any  program,  unstable  requirements  and  requirements  creep  have  the  potential  to 
increase  costs  dramatically.  A  strong  systems  engineering  process  that  emphasizes  thorough 
requirements  analysis  and  definition  will  minimize  the  impact  of  this  cost  driver. 

2.3.3  Cost  Guidance  for  WBS  Elements 

One  purpose  of  this  study  was  to  develop  a  robust  WBS  generic  across  a  broad  spectrum  of 
system  application  domains.  While  elements  in  the  proposed  WBS  are  relative  to  using 
distributed  testing  technology  (either  new  elements  or  those  that  would  be  modified  to 
accommodate  distributed  testing  technology  use),  other  elements  in  a  program’s  T&E  WBS  may 
be  deleted  or  significantly  reduced  with  distributed  testing  use. 
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The  following  tables  are  intended  to  provide  a  cost  analyst  with  initial  information  on  distributed 
testing  cost  elements.  Much  of  this  information  will  support  development  of  range  cost  estimates 
given  the  immature  state  of  relevant  cost  information  for  this  technology.  Included  are  cost 
elements  that  JADS  was  able  to  attribute  cost  information  and  recommendations  relevant  to  the 
development  of  a  cost  estimate  to  and  do  not  include  all  WBS  elements  defined  in  the  proposed 
WBS  in  Section  2.5. 

The  WBS  includes  only  those  elements  that  have  either  been  added  to  reflect  an  activity  new  to 
T&E  or  that  will  need  modification  because  of  inclusion  of  distributed  testing. 

The  JADS  experience  provided  only  three  limited  examples  from  which  this  information  has  been 
gleaned,  supplemented  by  information  from  other  sources.  This  information  may  not  be 
applicable  across  the  entire  T&E  spectrum.  As  stated,  this  effort  is  meant  to  provide  an  initial 
step  toward  developing  a  cost  model  that  reflects  distributed  testing  activities. 

Table  1.  Cost  Guidance,  Risks  and  Recommendations  for  WBS  Element  1.X.0 


WBS  Element 

Cost  Guidance  and  Risks 

Recommendations/Comments 

1  .X  System  Test  and  Evaluation 

1  .X.O  Feasibility  Analysis  (FA) 

The  FA  will  require  more  thorough  planning  and 
earlier  in  the  program.  JT&E  program  managers 
and  engineers  should  be  identified  and  work 
initiated  as  part  of  the  first  phase  of  the  programs 
life  cycle. 

Even  if  the  FA  results  in  a  decision  to  not  use  ADS 
technology,  it  will  provide  T&E  engineers  with 
needed  insight  into  the  system’s  requirements  and 
testing  needs  and  supplant  the  need  for  typical  initial 
studies. 

1  .X,0. 1  Test  Planning  Methodology 

Since  this  element  requires  in-depth  understanding 
of  the  SUT  requirements,  it  does  require  a  fair 
amount  of  effort.  However,  some  of  the  effort  done 
in  developing  this  product  could  support  either  other 
T&E  products  or  products  developed  for  the  overall 
program. 

Reference  the  JADS  special  report,  A  Test  Planning 
Methodology-From  Concept  Development 

Through  Test  Execution,  at  the  JADS  website. 

1  .X.0.2  ROM  Cost  Analysis 

Additional  costs  for  facilities,  networking  and 
development  and/or  modification  of  entities  for 
required  capabilities  should  be  included  in  the  cost 
estimate. 

A  range  estimate  at  this  stage  is  appropriate.  This 
approach  to  cost  analysis  will  provide  several 
attractive  uses.  First,  it  will  make  it  easier  to  arrive 
at  reasonable  costs  since  high,  medium  and  low 
value  estimates  are  sought,  not  an  absolute  value. 
Second,  it  can  support  quantification  of  risk  if  the 
estimated  values  are  identified  based  on  uncertainty. 

l.X.0.3  Risk  Analysis 

Quantifiable  risks  could  be  incorporated  into  the 

ROM  cost  analysis  through  use  of  a  range  cost 
estimating  approach.  Nonquantifiable  risks  should 
be  considered  and  potential  outcomes  understood  to 
the  extent  possible  for  this  early  stage.  Technical 
and  programmatic  risks  should  be  addressed. 

This  analysis  should  focus  on  two  areas:  first,  test 
requirements  that  cannot  be  met  using  traditional 
test  methods  and  second,  the  importance  of 
planning,  scheduling  and  coordination  of  federation 
members. 

25 


Table  2.  Cost  Guidance,  Risks  and  Recommendations  for  WBS  Element  l.X.1.1 


WBS  Element 

Cost  Guidance  and  Risks 

Recommendations/Comments 

l.X.  1  Developmental  Test  and 
Evaluation  (DT&E) 

This  stage  of  program  testing  typically  is  done  at  the 
brassboard  or  system  subcomponent  level. 

l.X.1.1  Planning 

Using  ADS  technologies  requires  increased 
planning  because  of  scheduling  and  time-critical 
linkage. 

Diligence  in  planning  and  stricter  adherence  to  good 
systems  engineering  practices,  while  necessary  for 
ADS  use,  will  enhance  the  results  of  the  entire  T&E 
program. 

1  .X.  1 . 1 . 1  T&E  Network  Requirement 
Definition 

Complexity  of  the  network  architecture  will  drive 
this  element’s  costs  up  since  there  will  be  more  to 
examine.  Developing  the  program  test  plan  will 
require  additional  effort  not  typically  done  for 
traditional  testing. 

Complexity  is  equated  to  number  of  nodes,  links 
and  required  bandwidth  to  develop  the  synthetic 
environment.  Complexity  will  drive  costs  of  the 
entire  T&E  test  program.  During  the  planning 
phase,  network  complexity  should  be  minimized  to 
the  extent  reasonably  possible. 

l.X.  1.1. 1.1  Requirements 

Identification 

Estimating  costs  for  identifying  requirements  can  be 
initially  started  using  DoD  CERs  for  this  activity 
and  should  then  be  modified  to  the  specific  program. 

1  .X.  1 . 1. 1 .2  Objectives  Development 

Complexity  of  the  network  architecture  will  drive 
costs  for  this  element. 

1  .X.  1 . 1.2  Establish  Federate 
Development  Team 

The  number  of  network  nodes  will  drive  costs  since 
entities  need  to  be  interacted  and  coordinated  with 
for  its  accomplishment. 

1.x.  1.1. 3  T&E  Phase  Reporting  and 
Documentation 

Documentation  will  be  similar  to  traditional  T&E. 

Use  the  same  cost  estimating  relationship  used  in  the 
rest  of  the  program  cost  analysis. 

Note,  this  element  includes  documentation  for  all 
DT&E  testing. 

1  .X.  1 . 1 .4  T&E  Phase  Maintenance 

T&E  phase  maintenance  costs  are  expected  to  be 
higher  than  traditional  T&E  methods  because  ADS 
technology  is  more  HW  and  SW  intensive.  Actual 
costs  can  be  estimated  based  on  the  bill  of  materials 
and  amount  of  SW  required  for  the  tests. 

While  these  costs  may  be  higher  than  for  traditional 
testing  HW  and  SW,  it  is  expected  that  this  cost  will 
be  more  than  offset  by  a  decrease  in  traditional  test 
costs  for  maintaining  live  test  equipment. 

l.X.1.1. 4.1  HW  Maintenance 

Use  vendor  maintenance  information. 

1  .X.  1 . 1 .4.2  SW  Maintenance 

DoD  CERs  for  software  maintenance  should  be 
applied  to  estimate  this  element’s  cost. 

Table  3.  Cost  Guidance,  Risks  and  Recommendations  for  WBS  Element  l.X.1.2 


WBS  Element 

Cost  Guidance  and  Risks 

Recommendations/Comments 

l.X.1.2  Concept  Development 

This  test  program  phase  will  require  significant 
coordination  effort  to  gain  consensus  on  scenario 
characteristics  and  test  objectives.  Effort  will  also 
be  used  for  federation  object  model  (FOM)  and 
interface  control  document  (ICD)  development  as 
well  as  the  test  concept. 

1  .X.  1 .2. 1  Scenario  Development 

Scenarios  are  not  limited  by  test  facility  or  range 
capabilities.  Additional  development  supports 
augmentation  of  these  capabilities  adding  significant 
capabilities  to  the  test  program  at  some  cost.  Costs 
are  a  variable  of  the  number  of  entities  required. 

JADS  ETE  Test  scenario  development  work  is  a 
good  example. 

1  .X.  1 .2.2  Concept  Analysis 

Based  on  JADS  experience,  costs  for  this  element 
are  more  than  for  traditional  testing  and  are  driven 
by  the  number  of  entities  that  must  be  included  in 
the  analysis. 
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Table  4.  Cost  Guidance,  Risks  and  Recommendations  for  WBS  Element  l.X.1.3 


WBS  Element 

Cost  Guidance  and  Risks 

Recommendations/Comments 

l.X.1.3  Design  and  Development 

Complexity  of  the  network  and  fidelity  of  the 
models  will  drive  the  costs  in  this  phase.  This  phase 
will  also  require  coordination  among  the  federates 
and  their  adherence  to  all  standards  and  protocols. 
Requirement  creep  or  change  will  cause  costs  and 
schedule  to  exceed  estimated  values. 

The  T&E  network  architecture  plans  and  all  plans 
for  development  and  implementation  are  completed. 
Development  and/or  procurement  of  HW  and  SW 
for  networks,  protocols,  interfaces  and  TCA  are 
accomplished. 

1  .X.  1 .3.1  T&E  Network  Architecture 
Design 

Costs  for  this  element  are  strongly  impacted  by  the 
number  of  nodes  included  in  the  federation.  The 

TCA  will  require  HW  and  SW  investments.  JADS 
TCAC  cost  about  $750  thousand  (K)  in  fiscal  year 
(FY)  97  dollars.  The  number  of  nodes  will  increase 
the  cost  of  developing  the  security  approach  since 
coordination  must  include  all  designated  approval 
authorities  (DAA). 

l.X.  1.3. 1.1  T&E  Network 

Architecture  Detailed  Design 

DoD  detailed  design  CERs  should  provide  a 
reasonable  starting  point  for  estimating  this  element. 
Modify  the  CER  as  seems  appropriate  to  the  case. 

l.X.  1.3. 1.1.1  Node  Site  Surveys 

Typical  CERs  can  be  used  for  temporary  duty 
expenses  to  visit  the  node  sites  once  they  are 
identified. 

Costs  are  driven  by  number  of  nodes. 

1  X.  1 .3. 1 . 1 .2  Test  Control  and 

Analysis  (TCA)  Definition 

JADS  TCAC  HW  and  SW  costs  are  approximately 
$750K  in  FY97  dollars. 

l.X.  1.3, 1.1. 3  Security  Approach 
Development 

Increased  requirements  and  coordination  with  each 
node’s  DAA. 

l.X.  1.3. 1.1.7  Test  Control  HW  and 

SW  Selection 

Moderately  increased  cost  to  provide  this  HW  and 

SW. 

l.X.  1.3. 1.1. 8  Data  Collection  and 
Instrumentation  Requirements 
Determination 

Tools  required  to  support  this  element  are  typically 
on  the  high  end  for  test  equipment. 

1  .X.  1 .3. 1 . 1 .9  Time  Synchronization 
Method 

It  is  important  that  the  federation  be  time 
synchronized  to  ensure  accurate  results. 

1X.1.3.1.1.10  T&E  Network 
Architecture  Development  and 
Integration  Plan 

Network  architecture  is  very  important  and  should 
be  designated  to  minimize  complexity,  e.g., 
minimum  set  of  nodes.  Architecture  integration 
must  be  well  planned. 

1  .X.  1 .3.2  Test  Activity  Planning 

This  activity  includes  some  of  the  W&A  activities. 
While  the  cost  of  this  element  will  vary  with  the 
number  of  nodes  required,  the  JADS  experience 
showed  that  this  was  not  a  significant  cost  to  the  test 
program. 

The  cost  of  this  element  will  vary  with  the  number 
of  nodes  required. 

l.X.  1. 3.2.2  Perform  Detailed  Design 
Verification 

Costs  driven  by  the  number  of  nodes. 

l.X.  1.3.3.  T&E  Network 

Architecture  Development 

JADS  had  additional  costs  because  of  lack  of  up-to- 
date  simulations  and  facilities.  To  contain  costs, 
leverage  DIS-  and  HLA-compliant  systems 
supporting  standard  data  protocols  and  data 
interfaces  to  avoid  modifications.  The  Foundation 
2010  Program  Office  is  currently  modernizing  five 
OARs  and  is  chartered  to  do  all  2 1  ranges.  DMSO 
HLA  office  cites  increasing  HLA  certified  systems. 
These  activities  should  support  cost  containment  for 
ADS  technology  users. 

Significant  time  should  be  spent  developing  this 
architecture  to  ensure  success  during  integration  and 
test  activities.  Test  ranges  (TR)  included  in  the 
federation  should  provide  detailed  cost  estimates. 

Cost  drivers  are  TR  staff  inexperienced  in  ADS 
technology  and  the  previous  task  having  little 
similarity  to  current  ADS  task.  JADS  experience 
with  TR  cost  estimates  was  that  they  were  10  to  15 
percent  lower  than  actual  costs. 

1X1. 3.3.3  HW  and  SW 

Configuration  Control  Implementation 

Costs  for  this  element  are  strongly  impacted  by  the 
number  of  nodes  included  in  the  federation  and 
changes  to  HW  and  SW  throughout  the  T&E  life 
cycle. 

Table  4.  Cost  Guidance,  Risks  and  Recommendations  for  WBS  Element  l.X.1.3  (continued) 


WBS  Element 

Cost  Guidance  and  Risks 

Recommendations/Comments 

l.X.1.3. 3.4  Data  Protocols 

Systems  that  leverage  DIS-  and  HLA-compliant 
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systems  avoid  developing  their  own  protocols. 

1 X  1.3. 3.5  Construct  and  Assemble 
Synthetic  Environment 

Designing  and  developing  software  for  player 
representations  are  a  major  cost  to  ADS  tests 
dependent  on  fidelity  and  complexity  (simple  script- 
driven  representations  to  man-in-the-loop 
simulators)  and  can  be  estimated  using  typical  SW 
estimating  techniques.  When  assembling  the  SE, 
number  of  nodes  and  architecture  complexity  drive 
the  costs. 

IX1.3.3.5.2  TCA  Design,  Build  or 
Procurement 

JADS  TCAC  costs  were  approximately  $750K  in 

FY  97  dollars. 

1  .X.  1 .3.3.6  Simulation  Modifications 

Simulation  costs  can  be  estimated  using  typical  SW 
estimating  techniques. 

Modifications  may  be  necessary  unless  the 
simulation  supports  standard  data  interfaces,  e.g., 

DIS,  HLA. 

1  .X.  1 .3.3.7  Range  Data  Processing 
Modifications 

ADS-based  tests  are  data  intensive.  Test  ranges 
should  provide  cost  estimates  for  required  data 
processing  upgrades. 

1  .X.  1 .3.3.8  Facilities  Modifications 

Facilities,  test  ranges  and  labs  should  be  asked  early  : 
in  T&E  planning  for  detailed  estimates  of  costs  to 
support  testing  that  clearly  lay  out  modification 
costs  the  program  must  bear. 

The  Foundation  2010  program  is  modernizing  five 
OARs  now  and  will  do  all  2 1  OARs.  Other 
facilities  are  getting  HLA  certified.  This  trend  will 
support  ADS  technology  making  it  affordable  for  all 
programs. 
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Table  5.  Cost  Guidance,  Risks  and  Recommendations  for  WBS  Element  l.X.1.4 


WBS  Element 

Cost  Guidance  and  Risks 

Recommendations/Comments 

l.X.1.4  Installation,  Integration 
and  Test  (II&T) 

Impact  on  costs  is  dependent  on  test  requirements 
and  is  driven  by  architecture  complexity  and 
required  fidelity.  ADS  is  HW  and  SW  intensive  and 
therefore,  costs  may  be  higher  for  II&T  than 
traditional  testing. 

1  .X.  1 .4. 1  Execution  Planning 

Use  typical  DoD  documentation  CERs  for  plan 
developments. 

Planning  needs  to  ensure  that  federates  are  all 
available  for  testing  and  synchronized.  Requires 
increased  coordination  with  all  federation  members. 

l.X.1.4. 1.4  Security  Test  and 
Evaluation  Plan 

Plan  may  need  to  address  multiple  security  levels  at 
varying  nodes. 

l.X.  1.4.2  Network  (NW)  Installation 
and  Testing 

Since  ADS  technology  is  more  network  based  than 
traditional  test  methodologies,  this  cost  element  can 
be  high. 

ADS -based  testing  requires  NW  monitoring  tools  to 
support  data  analysis.  These  tools  can  be  expensive, 
e.g.,  Cabletron’s  SPECTRUM®  SW  costs  were 
about  $25K  (depending  on  options).  Note,  this  is  a 
tool  for  NW  monitoring  and  not  the  actual  NW  SW. 

l.X.l.  4.2.1  HW  and  SW  Installation 

Since  ADS  technology  is  more  HW  and  SW 
intensive  than  traditional  test  methodologies,  this 
cost  element  may  be  higher. 

l.X.  1.4.2. 2  Integration  and  Testing 

This  element  should  be  driven  by  the  amount  of  HW 
and  SW  required  for  the  SUT  tests,  as  well  as  by  the 
fidelity  of  the  models  and  instrumentation 
requirements. 

1  .X.  1 .4.3  Components  and  Networks 
Integration 

The  cost  of  integrating  components  and  networks 
together  will  be  driven  by  the  architecture’s 
complexity. 

l.X.  1.4.4  Synthetic  Environment 
Validation 

The  cost  of  validating  the  SE  will  be  driven  by  the 
model’s  fidelity  requirement. 

1  .X.  1 .4.5  Risk  Reduction  Missions 

ADS-specific  cost;  necessary  activity  designed  to 
catch  problems  that  could  occur  during  test 
execution  causing  multiple  test  facilities  to  stay  in  a 
holding  pattern  or  possibly  loosing  the  facilities  to 
another  program’s  test. 

1  .X.  1 .4.6  Test  Environment 
Accreditation 

Low  cost  impact  relative  to  T&E  programs. 
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Table  6.  Cost  Guidance,  Risks  and  Recommendations  for  WBS  Elements 

l.X.1.5  and  1.X.2 


WBS  Element 

Cost  Guidance  and  Risks 

Recommendations/Comments 

l.X.1.5  Test  Execution  and 

Analysis 

ADS  test  execution  costs  could  be  considerably  less 
than  traditional  test  executions  depending  on  the 
architecture  complexity  and  fidelity  of  the  model 
versus  the  need  for  relocating  large  and  costly  test 
items.  OAR  facility  costs  are  expected  to  be  lower 
for  ADS  tests  than  traditional  tests. 

1  .X.  1 .5.1  Test  Readiness  Review 

Nominal  cost. 

1  .X.  1 .5.2  Test  Execution 

While  it  will  vary  with  the  architecture  complexity, 
ADS  test  execution  is  expected  to  cost  less  than 
traditional  test  execution.  ADS  testing  provides 
more  test  data  than  traditional  testing. 

l.X.l. 5.2.1  Test  Runs 

I.X.1. 5.2.2  Data  Collection 

Based  on  the  JADS  experience,  costs  for  data 
collection  may  be  about  10  percent  less  for  an  ADS- 
based  test  versus  a  traditional  test. 

1  .X.  1 .5.3  Results  and  Feedback 

ADS  testing  allows  for  more  scenarios  to  be  tested 
and  more  thorough  testing.  Better  testing  across  a 
wider  set  of  objectives  results  in  easier  evaluation 
and  reporting. 

Free  data  analysis  tools  are  available  from  the  JADS 
test  efforts. 

l.X.l. 5. 3.1  Data  Analysis 

Tools  required  for  data  analysis  are  divided  into 
real-time,  quick-look  and  detailed  analysis  tools. 

Due  to  availability  of  data  on  the  wide  area 
network,  tool  development  is  less  costly  than 
development  of  tools  for  non-ADS  tests  which 
requires  interfaces  to  various  median  and  formats. 
Tools  that  will  provide  a  majority  of  the  needed 
real-time  and  quick-look  analyses  are  available 
from  various  sources,  including  the  JADS  test 
efforts,  for  free.  The  reader  is  referred  to  Section 

2.6  of  this  report  for  detailed  information. 

l.X.l. 5. 3.2  Test  Evaluation 

Evaluation  of  ADS-based  tests  was  found  to  be 
easier  than  evaluation  of  traditional  tests  because  of 
the  amount  of  data  output  from  the  ADS  testing 
methodologies. 

1  .X.  1 .5.3.3  Provide  Test  Results  and 
Evaluation  Feedback 

This  element  should  be  aided  by  the  significant  data 
outputs  from  ADS  methods  and  should  cost  less 
than  traditional  test  efforts. 

1.X.2  Operational  Test  and 
Evaluation  (OT&E) 

OT&E  testing  demonstrates  a  system’s  ability  to 
support  the  services’  tactics,  techniques  and 
procedures. 

OT&E  is  likely  to  involve  more  nodes  than  DT&E 
resulting  in  higher  costs  relative  to  DT&E. 
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2.4  Proposed  Work  Breakdown  Structure  (WBS) 

This  section  presents  only  additional  or  modified  WBS  elements  identified  as  specific  to 
incorporation  of  distributed  testing  in  existing  T&E  approaches.  This  WBS  is  not  meant  to 
supplant  the  entire  T&E  WBS  but  only  to  provide  guidance  to  what  additional  elements  need  to 
be  inserted.  In  some  cases  these  elements  are  named  similarly  to  elements  in  WBS  for  existing 
T&E  approaches,  but  definitions  of  the  work  required  to  be  done  reflected  in  the  element  are 
directed  toward  the  use  of  distributed  testing. 

This  WBS  includes  tasks  required  to  incorporate  distributed  testing  as  part  of  the  T&E  process  of 
a  program.  It  is  not  a  stand-alone  product.  Elements  l.X.1.1  through  l.X.1.5.2  refer  to  specific 
activities  in  which  changes  to  conventional  T&E  approaches  will  have  to  be  considered  to  allow 
for  distributed  testing  use.  It  is  assumed  that  developmental  and  operational  tests  have  similar 
distributed  testing  requirements  and  costs.  Therefore,  this  report  provides  one  set  of  elements  in 
the  WBS  that  can  be  used  for  either  test  phase.  It  is  also  assumed  that  development  and 
implementation  of  the  distributed  testing  architecture  will  be  incorporated  into  the  T&E  process  in 
the  program’s  Phase  0  in  accordance  with  DoD  5000.2R. 

1.X  SYSTEM  TEST  AND  EVALUATION  (ST&E)  -  This  element  refers  to  the  use  of 
prototype,  production,  or  specifically  fabricated  HW  and  SW  to  obtain  or  validate  engineering 
data  on  the  performance  of  the  prime  mission  equipment  (PME).  This  element  includes  the 
detailed  planning,  execution,  support,  data  reduction,  and  reports  from  such  testing  (exclusive  of 
those  required  under  data),  and  all  HW  items  which  are  consumed,  or  planned  to  be  consumed,  in 
the  conduct  of  such  testing.  It  includes  all  effort  associated  with  the  design  and  production  of 
models,  specimens,  fixtures,  and  instrumentation  in  support  of  the  system-level  test  program.  It 
also  includes  all  planning,  management,  and  coordination  of  federation  developers  required  to 
incorporate  distributed  testing  use  into  traditional  T&E  methods.  Note  that  distributed  testing 
techniques  use  simulated  models  connected  via  real-time  networks.  Therefore,  the  term  T&E 
network,  or  T&E  network  components,  is  used  in  place  of  distributed  testing  in  most  cases  in  the 
following  WBS  elements  and  definitions. 

Initially,  it  was  planned  to  develop  separate  sections  of  the  WBS  for  DT&E  and  OT&E,  but  T&E 
engineers  described  similar  activities  for  both  test  phases.  This  WBS  provides  one  set  of  elements 
that  may  be  used  for  either  developmental  or  operational  testing.  For  a  new  project,  WBS 
element  1.X.0,  Feasibility  Analysis,  should  be  done  for  the  entire  T&E  program.  If  the  project  has 
already  gone  through  much  of  the  DT&E  phase,  1  .X.O  should  be  used  to  analyze  the  benefits  and 
utility  of  using  distributed  testing  technology  in  the  OT&E  phase. 

1.X.0  FEASIBILITY  ANALYSIS  -  The  purpose  of  this  element  is  to  provide  the  decision  maker 
(usually  a  program  manager)  with  necessary  information,  based  on  quantitative  and  qualitative 
analyses,  on  which  a  decision  can  be  made  to  use  or  not  use  distributed  testing  techniques  in  the 
program’s  T&E  phase.  Also,  if  distributed  testing  techniques  will  be  used,  this  analysis  should 
support  identification  of  how  it  can  best  be  used  to  supplement  traditional  T&E  capabilities  and 
what  resources  may  be  available  and  their  location. 
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l.X.0.1  TEST  PLANNING  METHODOLOGY  -  An  approach  to  analyzing  applicability  of 
distributed  testing  techniques  in  meeting  a  program’s  test  objectives.  Included  in  this  approach  is 
a  survey  of  availability  of  models  and  simulations,  networks,  facilities,  trained  personnel,  etc.,  to 
support  a  program’s  T&E  requirements.  This  analysis  should  also  address  the  program’s  required 
security  levels  and  ability  and  availability  of  equipment  and  personnel  who  can  sustain  the 
program’s  security  integrity.  Additional  information  on  this  methodology  can  be  found  in  the 
JADS  JTF  technical  paper,  “An  Advanced  Distributed  Simulation  Inclusive  Test  Planning 
Methodology.” 

l.X.0.2  ROM  COST  ANALYSIS  -  Perform  a  ROM  cost  estimate  analyzing  cost  of 
incorporating  distributed  testing  techniques  into  the  T&E  approach.  Costs  related  to 
development,  modification,  and/or  procurement  of  modeling  and  simulation  (M&S),  SW,  HW, 
networks,  facilities,  personnel,  training,  W&A,  and  documentation  are  some  examples  of 
elements  that  should  be  included  in  the  estimate. 

l.X.0.3  RISK  ANALYSIS  -  Perform  an  initial  analysis  of  technical  and  programmatic  risks  of 
incorporating,  or  not  incorporating,  distributed  testing  methods  into  the  traditional  T&E 
approach. 

This  cost  element,  1.X.0,  leads  to  a  decision  point  in  the  program  —  to  use  distributed  testing 
technology  with  traditional  methods  or  not.  Therefore,  it  is  essential  that  the  feasibility  analysis  be 
resourced  sufficiently  to  provide  solid  information  to  the  decision  makers.  If  the  decision  is  to  use 
distributed  testing  technology,  all  the  analysis  will  be  reused  and  refmed  in  the  T&E  planning 
phase.  If  the  decision  is  not  to  use  distributed  testing  technology,  it  is  expected  that  the  results  of 
the  analysis  would  still  be  of  use  in  the  T&E  phase. 

I.X.  1  DEVELOPMENTAL  TEST  AND  EVALUATION  (DT&E)  -  Test  and  evaluation  is 
conducted  on  systems  to  (a)  demonstrate  that  the  engineering  design  and  development  process  is 
complete,  (b)  demonstrate  that  the  design  risks  have  been  minimized,  (c)  demonstrate  that  the 
systems  will  meet  specifications,  (d)  estimate  the  system's  military  utility  when  introduced,  (e) 
determine  whether  the  engineering  design  is  supportable  (practicable,  maintainable,  and  safe)  for 
operational  use,  (f)  provide  test  data  with  which  to  examine  and  evaluate  trade-offs  against 
specification  requirements,  life  cycle  cost,  and  schedule,  and  (g)  perform  the  logistics  testing 
efforts  to  evaluate  the  achievement  of  supportability  goals;  the  adequacy  of  the  support  package 
for  the  system,  e.g.,  deliverable  maintenance  tools,  test  equipment,  technical  publications, 
maintenance  instructions,  and  personnel  skills  and  training  requirements,  etc.  DT&E  is  planned, 
conducted  and  monitored  by  the  developing  agency  of  the  DoD  component. 

I.X.  1.1  PLANNING  -  The  purpose  of  this  section  is  to  initiate  the  studies  and  analyses  of 
distributed  testing  requirements,  architecture,  resources,  and  constraints;  define  objectives;  and 
identify  critical  systems.  As  part  of  the  planning  effort,  organizations  that  will  be  used  in  the 
simulation  federation  should  be  contacted  and  talks  initiated  toward  achieving  the  required 
agreements  on  responsibilities  and  schedules.  Note  that  several  of  these  activities  will  continue 
definition  work  started  in  l.X.0.1. 
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l.X.  1.1.1  T&E  NETWORK  REQUIREMENTS  DEFINITION  -  Define  T&E  network  and 
components  test  objectives,  measures  of  effectiveness  and  performance,  needed  resources,  and 
constraints.  Initiate  development  of  plans  and  procedures  for  directing  architecture  and  tests 
toward  meeting  objectives. 

l.X.  1.1. 1.1  REQUIREMENTS  IDENTIFICATION  -  Develop  an  initial  problem  statement  from 
information  available  at  this  stage  of  development  and  generate  an  initial  requirements  list. 
Define  expected  W&A  requirements.  Information  may  come  from  white  papers,  position 
statements,  and  from  meeting  discussions. 

l.X.  1.1. 1.1.1  CRITICAL  SYSTEMS  OF  INTEREST  DESCRIPTION  -  Develop  a  clear  and 
unambiguous  statement  of  program  needs  to  include  a  high-level  description  of  critical  systems  of 
interest,  required  fidelity  and  resolution  of  simulated  entities,  and  output  data  requirements. 

l.X.  1.1. 1.1. 2  SUPPORT  RESOURCES  AVAILABILITY  -  Identify  resources  available  to 
support  the  T&E  network  architecture  (funding,  personnel,  tools,  facilities,  etc.). 

l.X.  1.1. 1.1. 3  PROGRAMMATIC  RISK  -  Determine  known  constraints  that  could  affect  the 
T&E  network  architecture  development,  e.g.,  due  dates,  and  security  requirements. 

l.X.  1.1. 1.1. 4  CONDUCT  W&A  FEASIBILITY  STUDY  -  Conduct  study  to  understand  the 
VV&A  requirements  and  identify  feasible  testing  requirements. 

l.X.  1.1. 1.1. 4.1  IDENTIFY  V&V  CONCEPT  -  Initial  conceptualization  of  the  V&V  approach 
and  requirements. 

l.X.  1.1. 1.1. 4.2  PERFORM  CONCEPTUAL  MODEL  V&V  -  Validate  the  conceptual  model 
against  the  T&E  network  components  requirements. 

l.X.  1.1. 1.1. 5  DEVELOP  PROGRAM  TEST  PLAN  -  Develop  the  test  schedule  and 

requirements  for  implementation  for  the  program. 

l.X.l. 1.1. 1.5.1  EXPAND  V&V  CONCEPT  -  Develop  additional  details  to  flesh  out  the  initial 
V&V  concept. 

I.X.1. 1.1. 1.5.2  IDENTIFY  COMPLIANCE  STANDARDS  STATUS  -  Identify  standards  and 
protocols  for  the  T&E  network  components. 

l.X.  1.1. 1.1. 5.3  PERFORM  ARCHITECTURAL  DESIGN  VERIFICATION  -  Using  the 
preliminary  design,  identify  information  (e.g.,  component  level  W&A  history,  fidelity 
characteristics)  about  network  components  that  can  assist  design  decisions.  Verify  the 
correctness  and  completeness  of  the  conceptual  model  or  preliminary  design. 
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l.X.  1.1. 1.2  OBJECTIVES  DEVELOPMENT  -  Refine  the  statement  of  program  needs  into  a 
more  detailed  set  of  specific  objectives  and  plans  for  the  T&E  network  architecture. 

l.X.  1.1. 1.2.1  TEST  OBJECTIVES,  SCENARIOS,  CONDITIONS  AND  MEASURES 
IDENTIFICATION  -  Develop  a  prioritized  list  of  measurable  objectives  for  the  T&E  network 
architecture.  Assess  availability  of  government  or  contractor  furnished  equipment,  facilities  and 
data.  Define  operational  context  constraints  or  preferences,  including  geographical  regions, 
environmental  conditions,  threats,  and  tactics. 

l.X.  1.1. 1.2.2  PLAN  DEVELOPMENT  -  Develop  the  T&E  network  architecture  and  network 
implementation  plans  showing  planned  schedule  and  major  milestones.  Develop  the  network 
configuration  management  plan  and  data  collection  test  plan. 

l.X.  1.1. 1.2.3  SECURITY  REQUIREMENTS  IDENTIFICATION  -  Identify  security  position, 
including  expected  security  level  and  possible  designated  approval  authorities. 

l.X.  1.1. 1.2.4  SUPPORT  TOOLS  SELECTION  -  Identify  and  select  tools  to  support  scenario 
development,  concept  analysis,  configuration  management,  W&A,  and  test  activities.  Tool 
selection  should  be  based  on  tool  availability,  cost,  maintainability,  applicability  to  the  given 
function,  and  the  ability  of  a  given  set  of  tools  to  exchange  data  within  the  T&E  network 
architecture. 

l.X.  1.1. 2  ESTABLISH  FEDERATE  DEVELOPMENT  TEAM  -  Designate  members  of 
federating  system’s  organizations  to  coordinate  development  of  distributed  testing  environment 
including  the  architecture,  federation  object  model  (FOM),  interface  control  document  (ICD), 
schedules,  and  other  test  requirements.  Define  requirements,  expectations,  test  objectives,  etc.,  to 
provide  these  organizations  with  an  understanding  of  what  resources  are  expected  from  them  in 
personnel,  training,  equipment,  and  what  costs  they  may  be  required  to  bear.  Develop  federate 
coordination  process  and  guidance. 

1  .X.  1 .1 .3  T&E  PHASE  REPORTING  AND  DOCUMENTATION  -  Document  and/or  report  on 
all  plans  and  procedures  required  achieving  the  objectives  of  the  elements  associated  with  1.X.1 
to  include  all  technical,  management,  programmatic  reports,  plans,  briefings,  engineering 
drawings,  etc. 

l.X.  1.1. 4  T&E  PHASE  MAINTENANCE  -  Includes  all  efforts  related  to  maintenance  of  T&E- 
specific  items:  planning,  contracting,  and  oversight. 

l.X.  1.1. 4.1  HARDWARE  MAINTENANCE  -  Includes  maintenance  effort  associated  with 
T&E-specific  HW  items. 

l.X.  1.1. 4.2  SOFTWARE  MAINTENANCE  -  Includes  maintenance  effort  associated  with  T&E- 
specific  SW  items. 
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l.X.  1.2  CONCEPT  DEVELOPMENT  -  The  purpose  of  this  section  is  to  develop  a 
representation  of  the  real-world  domain  of  interest  (entities  and  tasks)  in  terms  of  a  set  of  required 
objects  and  interactions. 

l.X.  1.2.1  SCENARIO  DEVELOPMENT  -  Develop  a  functional  specification  of  the  T&E 
network  test  scenario,  using  the  operational  context  constraints  identified  in  the  objectives 
development  (l.X.l. 1.1.2)  element. 

l.X.  1.2. 1.1  ENTITIES  IDENTIFICATION  -  Identify  all  entities  that  must  be  represented  in 
simulations,  or  otherwise,  by  the  T&E  network  architecture. 

l.X.  1.2. 1.2  FUNCTIONAL  DESCRIPTION  OF  ENTITIES  -  Document  functional  descriptions 
of  the  capabilities,  behavior,  and  relationships  between  entities  over  time. 

l.X.  1.2. 1.3  ENVIRONMENTAL  IMPACT  AND  CONDITIONS  STUDY  -  Identify  relevant 
environmental  conditions  that  impact  or  are  impacted  by  entities  in  the  T&E  network  architecture 
including  initial  and  terminal  conditions  (textual  scenario  descriptions,  event-trace  diagrams,  and 
graphical  illustrations  of  force  laydowns  and  communication  paths). 

I.X.  1.2. 1.4  CONCEPTUAL  MODEL  DEVELOPMENT  -  Select  technique(s)  for  developing  the 
conceptual  model  (e.g.,  static  process  flow  diagram,  process  flow  diagrams,  correlation  tables  of 
objects  and  activities,  descriptive  texts)  and  initiate  development  to  capture  all  features  of  the  test 
environment. 

l.X.  1.2. 1.5  FEDERATED  OBJECT  MODEL  (FOM)  DEVELOPMENT  -  Select  a  FOM 
development  approach  and  develop  the  FOM.  The  FOM  development  approach  should  be 
guided,  in  part,  on  the  federate  members,  for  example,  merging  individual  federate  simulation 
object  models  (SOMs),  starting  from  a  primary  federate  SOM  and  adding  objects  and  interactions, 
selecting  a  reference  FOM  and  modifying  it. 

l.X.  1.2. 1.6  INTERFACE  CONTROL  DOCUMENT  (ICD)  DEVELOPMENT  -  Addresses 
critical  aspects  of  the  data  including  byte  ordering,  precise  definition  and  process  for  coordinate 
transformations,  context  for  interpreting  data  and  all  other  information  necessary  to  create  the 
reference  test  condition  such  as  threat  rules  of  engagement,  the  aircraft  flight  profile,  threat 
locations,  and  a  test  condition  matrix  that  is  used  by  a  test  controller  to  activate  and  deactivate 
threats.  While  not  identified  as  a  product  on  the  FEDEP  checklist,  the  JADS  EW  Test  team 
found  the  ICD  to  be  a  key  document. 

l.X.  1.2.2  CONCEPT  ANALYSIS  -  Develop  an  implementation-independent  representation  of 
the  T&E  network  architecture. 

l.X.l. 2.2.1  ENTITY  CHARACTERISTICS,  RELATIONSHIPS  AND  INTERACTIONS 
EVALUATION  -  Identify  all  entity  characteristics  and  the  static  and  dynamic  relationships 
between  them.  Identify  behavioral  and  transformational  (algorithmic)  aspects  of  each  using  the 
T&E  network  architecture  scenario. 
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1X1. 2.2.2  ENTITIES  REPRESENTATION  -  Determine  entity  characteristics  (attributes)  and 
interaction  descriptors  (parameters). 

1X1. 2.2.3  T&E  NETWORK  ARCHITECTURE  REQUIREMENTS  DETERMINATION  - 
Determine  T&E  network  architecture  requirements  such  as  data,  latency,  synchronization, 
network,  interface,  test  control  and  monitoring. 

1X1.3  DESIGN  AND  DEVELOPMENT  -  The  purpose  of  this  section  is  to  reflect  effort 
required  for  the  detailed  design  of  the  test  environment  to  meet  the  test  objectives  and  ensure 
adequate  data  collection;  to  procure  commercial  off-the-shelf  (COTS)  and  developmental  HW, 
SW  and  special  purpose  interfaces;  and  to  finalize  schedules  for  installation,  integration,  W&A, 
and  test  execution. 

IX  1.3.1  T&E  NETWORK  ARCHITECTURE  DESIGN  -  Design  the  T&E  network 
architecture  required  to  test  and  evaluate  the  system. 

I.X.  1.3. 1.1  T&E  NETWORK  ARCHITECTURE  DETAILED  DESIGN  -  Design  a  federate¬ 
wide  systems  engineering  methodology  to  support  T&E  network  architecture  development  and 
integration.  This  requires  close  coordination  among  all  facilities  and  entity  participants  to  ensure 
a  common  understanding  of  the  T&E  network  architecture  goals  and  requirements. 

LX.  1.3. 1.1.1  NODE  SITE  SURVEYS  -  Select  T&E  network  nodes  and  survey  each  location. 
Node  selection  is  based  on  fidelity,  availability,  cost,  and  schedule.  Site  surveys  determine  facility 
communications  architecture  and  requirements  and  space  requirements  for  tester-supplied 
equipment  and  personnel. 

1X1.3. 1.1.2  TEST  CONTROL  AND  ANALYSIS  (TCA)  DEFINITION  -  Define  required 
TCA  capabilities  and  optimal  locations. 

1X1. 3. 1.1.3  SECURITY  APPROACH  DEVELOPMENT  -  Perform  security  risk  assessment 
and  develop  a  security  concept  of  operations. 

1X1.3. 1.1.4  T&E  WAN  REQUIREMENTS  DEFINITION  -  Define  T&E  wide  area  network 
(WAN)  requirements  including  bandwidth,  data  rate,  acceptable  latency,  network 
management/control  responsibility,  etc. 

1X1.3. 1.1. 5  T&E  NETWORK  COMPONENTS  DETERMINATION  -  Determine  if  DoD- 
sponsored  networks  meet  the  T&E  WAN  requirements  or  if  commercial  resources  are  required. 
If  DoD-sponsored  common-user  services  are  not  feasible,  application  for  waiver  from  the  Defense 
Information  Systems  Agency  (DISA)  and  contract  for  leased  lines  would  be  required. 

1X1.3. 1.1.6  T&E  NETWORK  HW  SELECTION  -  Select  HW  for  the  T&E  network  (routers, 
channel  service  units,  data  service  units,  multiplexers,  encryptors). 
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l.X.  1.3. 1.1. 7  TEST  CONTROL  HW  AND  SW  SELECTION  -  Select  test  control  HW  and  SW 
based  on  the  data  and  voice  communications  requirements  and  TCA  space  requirements  and 
layout. 

l.X.1.3. 1.1.8  DATA  COLLECTION  AND  INSTRUMENTATION  REQUIREMENTS 
DETERMINATION  -  Select  instrumentation  types  and  data  logging  SW  based  on  data 
requirements.  Determine  HW  requirements  for  data  storage  and  handling  (tape  drives,  compact 
disks,  optical  disks).  Develop  handling  procedures  for  collecting  and  storing  data  from  the 
distributed  network. 

l.X.  1.3. 1.1. 9  TIME  SYNCHRONIZATION  METHOD  DETERMINATION  -  Determine 
method  for  synchronizing  time  stamps  for  data  loggers.  Select  synchronization  HW  and  SW. 

1  .X.  1 .3 . 1 . 1 . 1 0  T&E  NETWORK  ARCHITECTURE  DEVELOPMENT  AND  INTEGRATION 
PLAN  -  Complete  final  version  of  formalized  plan  for  T&E  network  architecture  development 
and  integration. 

1  .X.l  .3.2  TEST  ACTIVITY  PLANNING  -  Identify  necessary  or  required  test  activities  for  V&V 
that  will  demonstrate  that  the  T&E  network  will  meet  its  requirements. 

l.X.1.3.2.1  DEVELOP  V&V  PLAN  -  Continue  refinement  of  the  V&V  concept  and  transition 
information  into  actionable  plan. 

l.X.  1.3. 2.2  PERFORM  DETAILED  DESIGN  VERIFICATION  -  Verify  the  T&E  network 
detailed  design  is  correct,  complete,  and  maintains  traceability  to  network  requirements. 

l.X.  1.3.3  T&E  NETWORK  ARCHITECTURE  DEVELOPMENT  -  Initiate  procurement, 
development,  and/or  modifications  of  tools  and  SW;  identify  procedures  to  build  the  T&E 
network  architecture.  Included  in  this  would  be  any  additional  player  representations  needed, 
DSMs,  HWIL  labs,  range  facilities  and  instrumented  live  assets,  and  special  purpose  interfaces 
(e.g.,  network  interface  units). 

l.X.  1.3. 3.1  T&E  NETWORK  INSTRUMENTATION  PROCUREMENT  -  Procure  or  develop 
network-specific  HW  tools  for  network  analysis  and  monitoring  including  network 
instrumentation. 

l.X.  1.3. 3. 2  SECURE  AND  ENCRYPTED  OPERATIONS  PROCEDURE  DEVELOPMENT 
AND  APPROVAL  -  Develop  secure/encrypted  operations  procedures  for  approval  by  the 
designated  approval  authority  (DAA).  Achieve  formal  security  accreditation  for  all  network  and 
facilities. 

l.X.  1.3. 3.3  HW  AND  SW  CONFIGURATION  CONTROL  IMPLEMENTATION  -  Implement 
strict  HW/SW  configuration  management  control. 
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l.X.  1.3. 3. 4  DATA  PROTOCOLS  DEVELOPMENT  -  Develop  standard  data  protocols  to 
satisfy  T&E  network  architecture  communications  and  interoperability  requirements. 

l.X.  1.3. 3. 5  CONSTRUCT  AND  ASSEMBLE  SYNTHETIC  ENVIRONMENT  -  Hook  federate 
networks’  HW  and  SW  items  together  (including  the  TCA)  to  form  the  T&E  synthetic 
environment. 

l.X.1.3.3.5.1  INTERFACE  DESIGN,  BUILD,  OR  PROCUREMENT  -  Design,  build,  or 
procure  simulation  and  network  interfaces,  runtime  infrastructure  (RTI)  interfaces  (if  required), 
and  special  purpose  interfaces. 

l.X.1.3.3.5.2  TCA  DESIGN,  BUILD,  OR  PROCUREMENT  -  Based  on  requirements,  design 
the  TCA  and  develop  or  procure  its  infrastructure. 

l.X.1.3. 3.5.3  PERFORM  COMPATIBILITY  VERIFICATION  -  Verify  compatibility  of 

network  components. 

l.X.  1.3. 3. 5.4  PERFORM  COMPLIANCE  STANDARDS  VERIFICATION  -  Test  network 
components  to  verify  they  communicate  adequately  using  the  required  protocol. 

l.X.1.3. 3.6  SIMULATION  MODIFICATIONS  -  Identify  and  implement  modifications  necessary 
to  use  external  inputs  and  to  generate  required  outputs  from  existing  simulations. 

l.X.1.3. 3.7  RANGE  DATA  PROCESSING  MODIFICATIONS  -  Identify  and  implement 
modifications  necessary  to  meet  time-space-position  information  accuracy,  smoothness,  and 
latency  requirements. 

l.X.l. 3.3.8  FACILITIES  MODIFICATIONS  -  Identify  and  implement  modifications  required 
for  replay  capability  to  be  used  during  integration  testing. 

l.X.  1.4  INSTALLATION,  INTEGRATION,  AND  TEST  -  The  purpose  of  this  section  is  to 
install  and  integrate  the  network  HW  and  SW  subcomponents  and  conduct  testing  to  ensure 
federates,  TCA,  and  network  architecture  meet  interoperability  requirements. 

l.X.  1.4.1  EXECUTION  PLANNING  -  Define  requirements  to  support  execution  of  the  T&E 
network  architecture. 

1  .X.  1 .4. 1 . 1  INTEGRATION  TEST  PLAN  -  Develop  integration  test  plan  to  support  incremental 
checks  on  configuration  during  T&E  network  architecture  build  up. 

l.X.  1.4. 1.2  TEST  CONTROL  PROCEDURES  -  Develop  test  control  procedures  for  T&E 
network  architecture  integration  and  test. 

l.X.  1.4. 1.3  DETAILED  TEST  EXECUTION  PLAN  -  Develop  detailed  test  execution  plan  for 
T&E  network  architecture  integration  and  test. 
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l.X.  1.4. 1.4  SECURITY  TEST  AND  EVALUATION  PLAN  -  Develop  security  test  and 
evaluation  plan  that  defines  procedures  for  testing  and  evaluating  network  security. 

I.X.  1.4.2  NETWORK  INSTALLATION  AND  TESTING  -  Integrate  all  participating  facilities 
and  entities  into  one  operating  environment  and  verify  that  all  components  are  interoperable  to  the 
degree  required  to  achieve  the  T&E  network  architecture  objectives. 

1  .X.  1 .4.2. 1  HW  AND  SW  INSTALLATION  -  Install  and  check  network  HW  and  SW. 

l.X.l. 4.2.2  INTEGRATION  and  TESTING  -  Integrate  all  participating  facilities  and  entities  into 
one  operating  environment  and  verify  that  all  components  are  interoperable  to  the  degree  required 
to  achieve  the  T&E  network  architecture  objectives. 

l.X.  1.4.3  COMPONENT  AND  NETWORKS  INTEGRATION  -  Integrate  all  components  and 
networks  together  in  a  synthetic  environment. 

l.X.  1.4.4  SYNTHETIC  ENVIRONMENT  VALIDATION  -  Examine  the  ability  of  the  network 
configuration  to  represent  the  intended  application’s  behavior,  appearance,  performance,  fidelity 
constraints,  and  interoperability  levels. 

l.X.  1.4. 5  RISK  REDUCTION  MISSIONS  -  Execute  scenarios  with  fully  linked  test  execution 
configuration.  Include  security  certification  (if  required)  and  evaluation  of  test  control  and 
monitoring  procedures,  as  well  as  data  collection  and  analysis  procedures. 

l.X.  1.4.6  TEST  ENVIRONMENT  ACCREDITATION  -  Based  on  review  of  the  accrediting 
authority  (i.e.,  sponsor)  the  T&E  network  architecture  is  accredited  for  its  intended  purpose.  An 
accreditation  decision  for  formal  acceptance  is  made. 

l.X.  1.5  TEST  EXECUTION  AND  ANALYSIS  -  The  purpose  of  this  section  is  to  conduct  the 
tests,  collect  data,  analyze  test  results,  and  provide  feedback  to  the  PM  and  developer. 

l.X.  1.5.1  TEST  READINESS  REVIEW  -  Review  T&E  network  architecture  to  assess  readiness 
for  formal  testing. 

l.X.  1.5.2  TEST  EXECUTION  -  Exercise  all  components  of  the  T&E  network  architecture  as  an 
integrated  whole  to  achieve  the  test  objectives. 

l.X.1. 5.2.1  TEST  REHEARSAL  AND  EXECUTION  -  Rehearse  and  execute  the  test;  generate 
data  according  to  the  test  plan  requirements. 

l.X.l. 5.2.2  DATA  COLLECTION  -  Collect  data  in  accordance  with  approved  data  collection 
procedures. 
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l.X.  1.5.3  RESULTS  AND  FEEDBACK  -  Analyze  test  execution  data  and  provide  report  to  PM 
and  developer. 

l.X.1.5.3.1  DATA  ANALYSIS  -  Analyze  outputs  from  the  test  execution  phase. 

l.X.  1.5. 3.2  TEST  EVALUATION  -  Evaluate  results  from  the  execution  phase  to  determine  if  all 
test  objectives  have  been  met.  If  all  objectives  were  not  met,  identify  corrective  actions  to 
implement  for  retest. 

I.X.1. 5.3.3  PROVIDE  TEST  RESULTS  AND  EVALUATION  FEEDBACK  -  Provide  test 
results  and  analyses  to  customer. 

1.X.2  OPERATIONAL  TEST  AND  EVALUATION  (OT&E)  -  This  element  addresses  test  and 
evaluation  conducted  by  agencies  other  than  the  developing  agency  to  assess  the  prospective 
system's  military  utility,  operational  effectiveness,  operational  suitability,  logistics  supportability 
(including  compatibility,  interoperability,  reliability,  maintainability,  and  logistics  requirements), 
cost  of  ownership,  and  need  for  any  modifications.  Initial  operational  testing  and  evaluation 
(IOT&E)  conducted  during  the  development  of  a  weapon  system  is  included  in  this  element.  This 
element  encompasses  such  tests  as  integrated  system  tests,  flight  tests  and  sea  trials,  mobility 
demonstrations,  on-orbit  tests,  spin  demonstrations,  stability  tests,  etc.,  and  support  thereto, 
required  to  prove  the  operational  capability  of  the  deliverable  system.  Contractor  support  (e.g., 
technical  assistance,  maintenance,  labor  material)  consumed  during  this  phase  of  testing  is  also 
included  in  this  element. 

If  distributed  testing  techniques  were  used  during  DT&E,  the  T&E  network  architecture  can  be 
reused.  If  modifications  are  required,  some  of  the  planning  and  preparation  elements  must  be 
revisited  prior  to  executing  the  test.  If  distributed  testing  techniques  were  not  used  for  DT&E,  all 
elements  from  l.X.2.1  through  l.X.2.5.3.3  must  be  incorporated  into  the  OT&E  effort. 
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3.0  Conclusion 


This  section  provides  links  to  more  information  on  implementing  distributed  testing  technology 
and  summarizes  the  report. 

3.1  Tools  and  Resources 

This  section  provides  information  on  tools  and  resources  that  support  use  of  distributed  testing 
technologies. 

Links  to  Internet  sites  are  provided  to  facilitate  further  investigation  of  the  use  of  distributed 
testing  technologies.  Most  of  the  sources  below  do  not  directly  address  the  T&E  phase,  but  do 
address  common  issues  associated  with  distributed  testing  use.  This  list  provides  points  of 
contact  at  several  DoD  websites  but  is  not  meant  to  be  exhaustive. 

•  Over  the  five-year  lifetime  of  the  JADS  JTF,  numerous  articles,  briefings,  reports,  manuals 
and  tools  have  been  developed  and  researched  in  support  of  the  three  JADS  tests  and  can  be 
found  at  http://www.iads.abq.com/html/iads/techpprs.htm.4 

•  Information  on  tools  can  be  found  at  http://www.iads.abq.eom/html/iads/techpprs.htm#tools.5 

•  Other  organizations  that  are  using  distributed  testing  technologies  and  that  might  be  used  in 
support  of  a  T&E  program  can  be  found  at 
http://www.iads.aba.eom/html/ads/engine.htm#who. 

•  AFSAA  has  been  employing  distributed  testing  technologies  for  a  variety  of  reasons  and  has 
been  a  proponent  of  their  use.  More  information  on  AFSAA  can  be  found  at 
http://www.afsaa.hq.af.mil/SAM/SAMT/. 

•  STRICOM  website  also  has  a  wealth  of  information  on  this  topic  at 
http://www.stricom.armv.mil/PRODUCTS/ADSTII/. 

•  DMSO’s  website  has  many  relevant  links  to  resources.  The  HLA  FEDEP  was  mentioned 
above.  Another  resource  is  the  Modeling  and  Simulation  Resource  Repository  (MSRR)  at 
http://www.msrr.dmso.mil/. 

3.2  Summary 

This  report  provides  PMs  with  findings,  conclusions,  and  lessons  learned  from  the  three  JADS’ 
tests  and  others  regarding  the  costs  and  benefits  of  using  distributed  testing.  Distributed  testing 
technologies  provide  new  capabilities  that  potentially  help  the  tester  overcome  limitations  with 
traditional  means  of  testing. 


4  After  1  March  2001  refer  requests  to  Headquarters  Air  Force  Operational  Test  and  Evaluation 
Center  History  Office  (HQ  AFOTEC/HO),  8500  Gibson  Blvd.  SE,  Kirtland  Air  Force  Base,  New 
Mexico  87117-5558,  or  Science  Applications  International  Corporation  (SAIC)  Technical 
Library,  2001  North  Beauregard  St.,  Suite  80,  Alexandria,  Virginia  22311. 

5  Ibid. 
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JADS  has  furthered  the  T&E  community’s  ability  to  incorporate  distributed  testing  technologies 
through  the  development  of  tools  and  methodologies  that  support  its  use.  It  has  championed  the 
use  of  this  technology  with  other  legacy  products  and  resources  that  help  mitigate  or  eliminate 
risk  encountered  by  PMs.  Using  both  distributed  testing  technology  and  traditional  test  methods 
in  a  T&E  program  provides  a  more  robust  and  complete  system  test.  Based  on  the  JADS 
experience  the  use  of  distributed  testing  could  not  only  improve  testing  but  can  reduce  costs. 
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5.0  Acronyms  and  Abbreviations 


A/C 

AASI 

ACETEF 

ADS 

ADST 

AFATDS 

AFB 

AFEWES 

AFOTEC 

AFSAA 

AIM 

AMRAAM 

ASAS 

ATACMS 

A-tracker 

C4I 

C4ISR 

CAMPS 

CD 

CDT&E 

CER 

CM 

COTS 

DAA 

DIS 

DISA 

DMSO 

DoD 

DSM 

DT 

DT&E 

ECM 

EDAC 

EMC 

EMI 

EOA 

ES 

ETE 


aircraft 

Advanced  Aircraft  Simulation  Interface 

Air  Combat  Environment  Test  and  Evaluation  Facility,  Patuxent  River,  Maryland; 
Navy  facility 

advanced  distributed  simulation 
advanced  distributed  simulation  technology 
Advanced  Field  Artillery  Tactical  Data  System 
Air  Force  Base 

Air  Force  Electronic  Warfare  Evaluation  Simulator,  Fort  Worth,  Texas;  Air  Force 
managed  with  Lockheed  Martin  Corporation 

Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force  Base,  New 
Mexico 

Air  Force  Studies  and  Analysis  Agency 
air  intercept  missile 

advanced  medium  range  air-to-air  missile 
All  Source  Analysis  System 
Army  Tactical  Missile  System 
automatic  tracker 

command,  control,  communications,  computers  and  intelligence 

command,  control,  communications,  computers,  intelligence,  surveillance  and 

reconnaissance 

Compartmented  All  Source  Analysis  System  Message  Processing  System 
compact  disk 

contractor  developmental  test  and  evaluation 

cost  estimating  relationships 

countermeasures 

commercial  off-the-shelf 

designated  approval  authority 

distributed  interactive  simulation 

Defense  Information  Systems  Agency 

Defense  Modeling  and  Simulation  Organization,  Alexandria,  Virginia 

Department  of  Defense 

digital  system  model 

developmental  test 

developmental  test  and  evaluation 

electronic  countermeasure 

MITRE  Corporation’s  Economic  and  Decision  Analysis  Center 

electromagnetic  capability 

electromagnetic  interference 

early  operational  assessment 

electronic  support 

JADS  End-to-End  Test 


44 


EW 

EXCIMS 

FA 

FEDEP 

FOFSD 

FOM 

FOT&E 

FY 

GPS 

HITL 

HLA 

HW 

HWIL 

ICD 

IIT 

IOT&E 

ISTF 

JADS 

Janus 

JMASS 

Joint  STARS 

JT&E 

JTF 

K 

LFP 

LGSM 

LSP 

M&S 

MISILAB 

MITRE 

MOT&E 

MS 

MSRR 

NIU 

NW 

OA 

OAR 

OFP 

OT 

OT&E 

OUSDA(A&T) 

PDU 

PGM 

PM 

PME 


electronic  warfare:  JADS  Electronic  Warfare  Test 
Executive  Council  for  Modeling  and  Simulation 
feasibility  analysis 

federation  development  and  execution  process 
follow-on  full  scale  development 
federation  object  model 
follow-on  operational  test  and  evaluation 
fiscal  year 

global  positioning  system 

hardware-in-the-loop  (electronic  warfare  references) 

high  level  architecture 

hardware 

hardware-in-the-loop  (system  integration  references) 
interface  control  document 
installation,  integration  and  test 
initial  operational  test  and  evaluation 
installed  systems  test  facility 

Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 

interactive,  computer-based  simulation  of  combat  operations 

Joint  Modeling  and  Simulation  System 

Joint  Surveillance  Target  Attack  Radar  System 

joint  test  and  evaluation 

joint  test  force 

thousand 

live  fly  phase 

light  ground  station  module 
linked  simulators  phase 
modeling  and  simulation 

Missile  Simulation  Laboratory,  Eglin  Air  Force  Base,  Florida 
company  that  provides  engineering  services 
multiservice  operational  test  and  evaluation 
milestone 

Modeling  and  Simulation  Resource  Repository 

network  interface  unit 

network 

operational  assessment 
open  air  range 
operational  flight  program 
operational  test 
operational  test  and  evaluation 

Office  of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology) 

protocol  data  unit 

precision  guided  munitions 

program  manager 

primary  mission  equipment 
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PMO 

program  management  office 

QT&E 

qualification  test  and  evaluation 

RF 

radio  frequency 

ROM 

rough  order  of  magnitude 

RPSI 

radar  processor  simulator  and  integrator 

RTI 

runtime  infrastructure 

RWS 

remote  workstation 

SAIC 

Science  Applications  International  Corporation 

SBA 

Simulation  Based  Acquisition 

SE 

synthetic  environment 

SEPM 

systems  engineering  and  program  management 

SIL 

system  integration  laboratory 

SIT 

JADS  System  Integration  Test 

SOM 

simulation  object  model 

SPECTRUM® 

a  network  analysis  package  developed  by  Cabletron  Systems 

SPJ 

self-protection  jammer 

ST&E 

system  test  and  evaluation 

STEP 

simulation,  test  and  evaluation  process 

STRICOM 

U.S.  Army  Simulation,  Training  and  Instrumentation  Command 

SUT 

system  under  test 

SW 

software 

T&E 

test  and  evaluation 

TAC 

target  analysis  cell 

TADIL 

tactical  digital  information  link 

TAFSM 

Tactical  Army  Fire  Support  Model 

TBA 

theater  battle  arena 

TCA 

test  control  and  analysis 

TCAC 

JADS  Test  Control  and  Analysis  Center,  Albuquerque,  New  Mexico 

TDP 

Time-Space-Position  Information  Data  Processor 

TDY 

temporary  duty 

TJU 

tactical  digital  information  link  (TADIL)  J  upgrade 

TR 

test  ranges 

TRAC 

U.S.  Army  Training  and  Doctrine  Command  Analysis  Center 

TSLA 

Threat  Simulator  Linking  Activity 

TSPI 

time-space-position  information 

V&V 

verification  and  validation 

VSTARS 

Virtual  Surveillance  Target  Attack  Radar  System 

VV&A 

verification,  validation  and  accreditation 

WAN 

wide  area  network 

WBS 

work  breakdown  structure 

WSMR 

White  Sands  Missile  Range,  New  Mexico 
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Appendix  A 

Cost  Benefit  Analysis  for  the  System  Integration  Test 

A  study  performed  by  the  JADS  System  Integration  Test  team 

A.1.0  Introduction 

The  System  Integration  Test  (SIT)  Linked  Simulators  Phase  (LSP)  and  Live  Fly  Phase  (LFP) 
distributed  testing  architectures  provided  valid  results  for  air-to-air  missile  test  and  evaluation 
(T&E)  and  have  been  judged  to  have  utility  for  several  types  of  missile  performance  and  launcher 
integration  evaluations.  Because  of  the  high  cost  of  expending  missiles  and  destroying  target 
drones  in  live  fire  tests,  a  high-fidelity,  nondestructive  distributed  test  of  the  missile  has  the 
potential  to  save  significant  money,  especially  if  distributed  testing  is  utilized  over  several  phases 
of  test. 

This  appendix  lays  out  a  general  methodology  for  determining  the  cost  effectiveness  of 
implementing  distributed  testing  in  a  precision  guided  munitions  (PGM)  test  program  and 
provides  a  cost  savings  analysis  example.  Two  approaches  to  cost  effectiveness  are  addressed: 
(1)  a  cost  savings  approach  and  (2)  a  more  effective  test  approach.  For  the  first  approach,  total 
testing  costs  are  reduced  by  replacing  a  limited  number  of  live  fire  tests  with  distributed  tests.  In 
the  second  approach,  tests  are  added  without  eliminating  any  live  fire  tests,  so  that  the  total 
testing  costs  are  higher  but  with  some  compensating  qualitative  benefits. 

A.2.0  General  Methodology 

The  general  cost-effectiveness  methodology  focuses  on  qualitative  benefits  as  well  as  quantitative 
cost  factors.  The  outline  of  the  methodology  is  as  follows. 

-  Analyze  the  appropriate  use  of  distributed  testing  in  the  total  acquisition  and  testing  program. 

-  Estimate  the  potential  benefits  of  implementing  distributed  testing. 

-  Determine  any  probable  testing  constraints  that  might  limit  distributed  testing 
implementation. 

-  Estimate  costs  with  and  without  distributed  testing  implementation. 

-  If  distributed  testing  results  in  lower  testing  costs,  quantify  cost  savings  and  assess  benefits 
and  deficits  of  distributed  testing  use.  Significant  cost  savings  may  outweigh  slight 
deficits. 

-  If  distributed  testing  costs  about  the  same,  determine  the  benefits  or  deficits  of  using 
distributed  testing. 

-  If  distributed  testing  results  in  higher  total  testing  costs,  determine  if  the  benefits  justify 
the  increased  costs  or  if  other  program  costs  can  be  traded  off. 

-  Structure  an  optimal/near  optimal  distributed  testing  program  based  on  the  appropriate 
balance  between  cost  savings  and  qualitative  benefits. 
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The  following  subsections  describe  each  of  the  methodology  steps  in  more  details. 

A.2.1  Appropriate  Use  of  Distributed  Testing 

The  general  role  of  distributed  testing  in  PGM  T&E  was  discussed  in  Section  3.2.3  of  the  JADS 
Utility  of  Advanced  Distributed  Simulation  For  Precision  Guided  Munitions  Testing  report. 
Table  1  from  that  report  identified  the  various  performance  evaluation  techniques  used  for  PGM 
T&E  including  distributed  tests.  Although  the  discussion  in  that  section  focused  on  the  role  of 
distributed  testing  in  PGM  developmental  test  and  evaluation  (DT&E)  and  operational  test  and 
evaluation  (OT&E),  distributed  testing  can  support  the  other  testing  phases  including  early 
operational  assessment  (EOA),  operational  assessment  (OA),  initial  operational  test  and 
evaluation  (IOT&E),  and  follow-on  operational  test  and  evaluation  (FOT&E).  This  is  illustrated 
in  Figure  Al. 
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Figure  Al.  Role  of  Distributed  Testing 
During  PGM  System  Acquisition  and  Testing  Life  Cycle 


As  Figure  Al  shows,  a  PGM  digital  system  model  (DSM)  normally  becomes  available  during  the 
initial  system  acquisition  phase,  so  distributed  testing  evaluations  that  use  a  PGM  DSM  can  begin 
during  this  phase  (e.g.,  for  requirements  development),  even  before  formal  T&E  begins.  As  soon 
as  prototype  PGM  hardware  is  developed  during  the  second  acquisition  phase,  all  T&E  methods 
can  begin  to  be  used,  as  appropriate.  As  system  acquisition  proceeds,  all  T&E  methods  can 
continue  to  be  used  to  test  the  evolving  PGM  system  and  subsystems.  The  optimal  use  of  the 
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various  methods  depends  on  the  PGM  performance  areas  being  evaluated  as  Table  2  in  Section 
3.2.3  in  the  JADS  PGM  report  shows.  Note  from  Figure  A1  above  that  distributed  testing  can  be 
used  throughout  the  system  acquisition  and  testing  life  cycle  as  PGM  simulation  resources  are 
developed  and  refined.  Considerations  for  applying  distributed  testing  to  PGM  T&E  are 
discussed  in  Section  4  of  the  JADS  PGM  report. 

A.2.1.1  Distributed  Testing  Benefits 

In  analyzing  the  appropriate  use  of  distributed  testing  in  the  total  testing  program,  the  potential 
benefits  of  adding  distributed  testing  should  be  evaluated.  General  benefits  are  discussed  in 
Section  3.2.2  of  the  JADS  PGM  report  and  fall  into  the  categories  of  cost  savings,  improved 
testing,  and  more  efficient  testing.  Potential  cost  savings  are  determined  in  the  next  step,  but  any 
benefits  from  improved  or  more  efficient  testing  should  be  considered  before  proceeding.  These 
are  repeated  from  Section  3.2.2  of  the  JADS  PGM  report  and  expanded. 

-  Improved  testing  benefits. 

-  Testing  using  a  linked  laboratory  distributed  testing  architecture  is  more  reproducible  than 
live  fire  testing  because  scenario  conditions  are  more  readily  controlled  and  trials  can  be 
replayed  for  additional  PGM  responses.  This  allows  more  trials  to  be  combined  for 
analysis,  giving  greater  confidence  in  evaluation  results. 

-  Testing  using  manned  shooter  and  target  simulators  linked  to  a  PGM  DSM  or  hardware- 
in-the-loop  (HWIL)  laboratory  can  be  used  to  realistically  evaluate  man-in-the-loop 
reactive  countermeasures  (CM).  This  cannot  be  done  in  live  fire  testing  because  of 
obvious  safety  constraints. 

-  Distributed  testing  allows  the  evaluation  of  certain  classified  techniques  in  which  the 
electronic  countermeasures  (ECM)  device  cannot  be  permitted  to  radiate  its  radio 
frequency  (RF)  emission  on  an  open  range.  Rather,  the  ECM  emissions  can  be  restricted 
to  the  PGM  HWIL  laboratoiy  where  they  are  screened  from  unauthorized  observation  and 
where  analysts  can  immediately  observe  the  effects  of  the  ECM  on  PGM  performance. 

-  Distributed  testing  allows  the  force  density  of  the  scenario  to  be  increased.  The  number  of 
friendly  and  threat  systems  can  be  increased  by  representing  them  with  either  manned 
laboratories  (if  realistic  man-in-the-loop  control  of  the  systems  is  needed)  or  DSMs  (if 
scripted  behavior  is  acceptable).  The  inability  to  evaluate  system  performance  in  combat- 
representative  environments  is  a  common  limitation  in  OT&E  and  an  area  in  which 
distributed  testing  can  improve  the  operational  test  (OT)  environment. 

-  Distributed  tests  exhibit  more  realism  than  either  analytical  simulation  models  (because 
actual  hardware  is  used)  or  stand-alone  HWIL  laboratories  (because  realistic  shooter  and 
target  inputs  are  provided). 

-  The  live  shooter-target  distributed  testing  architecture  can  be  used  for  realistic 
engagement  tactics  development  and  evaluation. 

-  More  efficient  testing  benefits. 

-  Shooter/PGM  integration  can  be  effectively  evaluated  using  distributed  testing 
configurations. 

-  Integration  check-out  can  begin  early  in  the  acquisition  cycle  before  a  complete  PGM 
system  is  available  by  using  the  linked  laboratory  distributed  testing  configuration  and 
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including  only  key  PGM  subsystems  in  the  PGM  HWIL  laboratory.  Note  that  this 
same  configuration  can  also  be  used  to  verify  the  integration  of  a  new  shooter  platform 
to  an  existing  PGM  system. 

-  If  the  PGM  receives  data  link  support  from  the  shooter  during  its  flyout,  the  live 
shooter-target  distributed  testing  configuration  can  be  used  to  accurately  evaluate  the 
quality  of  the  support  messages. 

-  Testing  using  a  live  shooter-target  distributed  testing  architecture  is  more  efficient  than 
live  fire  testing  because  the  analysts  get  immediate  feedback  on  each  trial  of  a  multiple  trial 
mission.  This  allows  adjustments  to  be  made  to  the  remaining  test  matrix,  if  necessary, 
while  the  live  shooter  and  target  platforms  are  still  on  range.  This  “analyst-in-the-loop” 
feature  of  distributed  testing  would  be  especially  useful  in  efficiently  progressing  through 
an  ECM  testing  matrix  which  involves  varying  a  number  of  ECM-related  parameters.  (Up 
to  15  trials  were  executed  during  each  two-hour  SIT  LFP  mission.) 

-  Live  fire  tests  can  be  realistically  rehearsed  using  distributed  testing.  This  would  ensure 
the  proper  setup  of  the  scenario  and  reduce  wasted  live  fire  attempts  in  which  the  proper 
scenario  conditions  are  not  achieved  or  would  result  in  anomalies  (i.e.,  cost  avoidance). 
This  use  of  distributed  testing  would  also  reduce  the  risk  of  a  live  fire  testing  program  by 
identifying  scenarios  which  cannot  be  correctly  executed  or  which  cannot  achieve  the 
stated  objectives. 

Also,  distributed  testing  implementation  may  benefit  other  parts  of  the  PGM  acquisition  program 
besides  testing.  Distributed  testing  using  a  DSM  to  represent  the  PGM  may  aid  in  requirements 
development,  and  resources  developed  during  distributed  testing  implementation  may  be  useful 
for  training.  Such  benefits  of  distributed  testing  implementation  that  are  beyond  the  normal  scope 
of  testing  should  also  be  considered  in  this  determination. 

A.2.1.2  Distributed  Testing  Implementation  Constraints 

Constraints  and  limitations  in  applying  distributed  testing  must  be  considered  when  determining 
the  appropriate  use  of  distributed  testing.  These  include  any  constraints  in  using  the  candidate 
PGM  simulation  being  considered  for  linking  and  any  distributed  testing-related  constraints  as 
discussed  in  Section  3.3  of  the  JADS  PGM  report.  The  result  of  considering  these  constraints 
will  be  a  determination  of  which  PGM  test  scenarios  and  conditions  can  be  adequately  evaluated 
using  distributed  testing  (i.e.,  which  scenarios/conditions  are  feasible  candidates  for  distributed 
testing). 

A.2.2  Test  Program  Costing 

As  discussed  in  Section  3.2.2  of  the  JADS  PGM  report,  the  benefits  of  distributed  testing  are  best 
realized  when  this  technique  is  added  to  a  total  PGM  testing  program  and  are  used  to  supplement, 
rather  than  replace,  other  testing  techniques.  Therefore,  a  total  testing  program  should  be 
developed  with  and  without  distributed  testing.  Major  cost  elements  for  each  testing  technique 
are  as  follows 

-  Nonrecurring  costs.  Each  testing  technique  will  require  development  and  verification 
before  PGM  testing  can  begin.  The  cost  of  implementing  distributed  testing  can  be 
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significantly  reduced  if  distributed  testing  is  planned  for  as  the  other  testing  resources  are 
developed  (so  that  no  major  modifications  of  other  resources,  such  as  HWIL  laboratories, 
will  be  needed  for  distributed  testing  implementation  and  interface  costs  will  be 
minimized). 

-  Live  fire  tests.  The  most  significant  recurring  costs  are  for  the  PGM,  target  drone,  and 
test  range  support. 

-  Captive-carry  tests.  The  main  recurring  cost  is  for  test  range  support  including  use  of  the 
shooter  and  target  platforms  and  the  captive-carry  PGM  pod. 

-  Stand-alone  HWIL  tests.  The  main  recurring  cost  is  for  HWIL  facility  operation. 

-  Distributed  testing  using  live  shooter-target  configuration.  The  recurring  costs  combine 
the  cost  of  a  captive-carry  mission  and  stand-alone  HWIL  tests  with  the  cost  of  any 
additional  real-time  processing  of  the  data  transferred  from  the  live  platforms  to  the  PGM 
HWIL  laboratory.  As  a  result,  this  type  of  distributed  test  will  usually  cost  more  than  just 
a  captive-carry  mission  combined  with  a  stand-alone  HWIL  test  in  which  data  captured 
from  the  captive-carry  mission  are  replayed  to  drive  the  HWIL  laboratory  post-mission. 

-  Distributed  test  using  linked  HWIL  laboratories.  The  recurring  costs  will  be  the  sum  of 
the  costs  for  operating  each  laboratory  plus  the  cost  of  any  additional  real-time  processing 
of  the  transferred  data. 

The  cost  elements  are  combined  with  proposed  testing  programs  which  either  implement  or  do 
not  implement  distributed  testing  in  order  to  determine  the  total  costs  of  the  two  candidate 
programs  (with  and  without  distributed  testing). 

A.2.2.1  Cost  Savings  Case 

The  largest  potential  cost  savings  occur  when  distributed  tests  are  used  to  replace  some  of  the  live 
fire  tests.  Typically,  the  cost  of  the  PGM  and  the  target  drone  are  much  more  than  the  range 
support  cost  (e.g.,  the  Advanced  Medium  Range  Air-to-Air  Missile  [AMRAAM]  missile  and  QF- 
106  drone  cost  about  $550,000  compared  to  about  $50,000  for  range  support).  The  replacement 
of  a  live  fire  test  with  distributed  testing  essentially  saves  the  cost  of  the  PGM  and  target,  since 
the  range  support  cost  for  the  live  test  is  nearly  the  same  as  the  recurring  cost  of  the  distributed 
test. 

However,  there  are  limitations  in  eliminating  live  fire  tests,  because  public  law  requires  some  live 
PGM  testing  and  because  usually  a  minimum  number  of  live  shots  are  needed  to  evaluate  the 
PGM  reliability.  If  the  number  of  live  shots  is  already  at  the  minimum  needed  for  the  reliability 
evaluation,  then  none  of  the  live  shots  can  be  replaced  with  distributed  testing.  In  this  case,  the 
blending  of  distributed  testing  into  the  PGM  testing  program  will  normally  result  in  increased  cost 
(unless  the  distributed  tests  can  replace  some  of  the  nondistributed  simulation  testing  at  no 
additional  cost),  and  the  benefits  of  using  distributed  tests  must  be  carefully  evaluated  against  the 
increased  costs. 

If  some  of  the  live  fire  tests  can  be  replaced  with  distributed  testing  without  impacting  other 
testing  requirements  (e.g.,  the  reliability  determination),  then  the  maximum  allowable  number  of 
replacements  is  determined.  The  test  matrix/scenarios/conditions  are  then  examined  to  determine 
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which  of  these  are  best  accomplished  using  live  fire  tests  and  which  can  be  adequately  tested  using 
distributed  testing  (see  Section  B.2.1.1)  up  to  the  maximum  allowable  number  of  replacements. 
Note  that  each  live  fire  test  can  only  evaluate  a  single  scenario,  but  that  each  distributed  test 
mission  can  usually  evaluate  multiple  scenarios  (e.g.,  during  LFP  testing,  a  single  two-hour 
mission  consisted  of  about  15  separate  passes,  each  of  which  could  have  used  a  separate 
profile/scenario)  or  repeat  the  same  scenario  multiple  times  for  added  confidence.  Hence,  there 
can  be  considerable  leverage  in  replacing  live  shots  with  distributed  tests  (more  test 
conditions/trials  can  be  evaluated  at  a  lower  cost). 

A.2.2.2  No  Cost  Savings  Case 

For  testing  programs  in  which  no  live  fire  tests  can  be  eliminated,  it  may  be  possible  to  add 
distributed  testing  without  increasing  the  total  testing  costs.  Replacing  some  of  the  tests  using 
stand-alone  simulation  techniques  with  distributed  tests  would  do  this.  As  discussed  in  Section 
3.2.3  of  the  JADS  PGM  report,  there  are  some  cases  in  which  distributed  tests  allow  PGM 
performance  areas  to  be  evaluated  which  could  not  be  evaluated  using  the  PGM  simulation  in  a 
stand-alone  configuration.  This  is  especially  true  for  those  cases  in  which  it  is  necessary  to 
provide  man-in-the-loop  shooter  and  target  inputs  to  the  PGM  simulation. 

In  general,  each  distributed  test  will  cost  more  than  the  corresponding  stand-alone  simulation  test. 
Thus,  in  order  to  keep  the  total  testing  costs  constant,  it  will  be  necessary  to  only  replace  a 
fraction  of  the  stand-alone  tests,  and  the  distributed  test  cases  must  be  carefully  selected  to  be 
only  those  that  require  linking  for  valid  results. 

Even  in  those  cases  in  which  some  live  fire  tests  can  be  replaced  with  distributed  tests,  the 
preference  may  be  to  keep  the  total  testing  costs  constant  by  adding  the  appropriate  number  of 
distributed  missions  to  replace  each  live  test.  This  would  allow  many  more  test 
scenarios/conditions  to  be  evaluated  during  the  testing  program.  For  example,  results  from  the 
SIT  LFP  show  that  when  the  cost  of  an  AMRAAM  missile  and  a  QF-106  drone  are  considered,  a 
single  live  AMRAAM  live  fire  test  could  be  replaced  by  about  ten  distributed  tests  that  use  the 
LFP  distributed  testing  architecture  at  no  additional  cost.  Further,  each  of  these  two-hour 
distributed  testing  missions  can  normally  execute  about  15  passes.  Hence,  for  this  example,  it 
would  be  possible  to  trade  off  one  live  fire  test  profile  against  150  distributed  test  profiles.  The 
result  would  be  greater  confidence  in  the  results  of  the  PGM  evaluation. 

A.2.2.3  Cost  Increase  Case 

When  distributed  tests  are  merely  added  to  a  total  testing  program  without  replacing  any  of  the 
other  testing  techniques,  the  total  testing  costs  will  increase.  In  this  case,  as  in  the  no  cost 
increase  case,  it  is  necessary  to  carefully  select  test  scenarios/conditions  for  distributed  testing 
implementation  and  to  thoroughly  evaluate  the  added  benefits  of  using  distributed  testing  versus 
the  increased  costs.  A  potential  benefit  that  should  be  considered  by  the  PGM  program  manager 
is  cost  avoidance  because  of  reduced  risk.  If  improved  testing  in  any  program  phase  would  result 
in  a  significant  reduction  in  the  risk  of  redesign  or  redevelopment,  the  program  manager  could 
trade  off  the  cost  avoidance  against  the  investment  in  distributed  testing. 
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Using  distributed  testing  would  not  necessarily  result  in  an  increase  in  the  total  cost  of  the  PGM 
acquisition  program  in  this  case.  Because  of  the  benefits  of  adding  distributed  testing,  the  PGM 
program  manager  should  consider  trading  off  other  program  costs  (e.g.,  those  for  requirements 
definition  or  training,  since  distributed  testing  configurations  may  be  able  to  assist  in  these  areas) 
in  order  to  implement  distributed  testing  without  increasing  the  total  program  cost. 

A.2.3  Optimal  Distributed  Testing  Program 

After  completing  the  analysis  in  the  previous  methodology  step,  it  should  be  straightforward  to 
construct  the  optimal  distributed  testing  program.  Cost  savings,  qualitative  benefits,  or  a 
combination  of  these  would  justify  the  resulting  program. 

A.3.0  Cost  Benefit  Examples 

A.3.1  Cost  Elements  Example 

Data  were  collected  during  the  SIT  LFP  for  the  costs  of  implementing  the  LFP  distributed  testing 
architecture;  conducting  AMRAAM  testing  using  this  architecture;  and  conducting  AMRAAM 
testing  using  traditional,  nondistributed  testing  techniques.  The  approximate  costs  for  executing 
each  testing  technique  are  as  follows  (test  planning  and  data  analysis  costs  are  not  included). 

•  Live  fire  test  cost:  about  $600,000  including  the  cost  of  the  AMRAAM  and  target  drone 
($600,000  per  pass) 

•  Captive-carry  flight  test  cost:  about  $24,000  for  a  two-hour  mission  that  can  perform  15 
passes  ($1600  per  pass) 

•  Stand-alone  HWIL  mission  cost:  about  $7,000  for  a  two-hour  mission  that  can  nominally 
perform  25  runs  ($280  per  run) 

•  LFP  distributed  testing  mission  cost:  about  $44,000  for  a  two-hour  mission  that  achieves  15 
passes  (~$2900  per  pass) 

•  LFP  distributed  testing  implementation  cost:  about  $2  million  including  the  cost  of 
developing  the  special-purpose  Advanced  Aircraft  Simulation  Interface  (AASI), 
modifying/upgrading  Eglin  Air  Force  Base,  Florida,  range  capabilities  and  facilities,  and 
integration  testing.  Note  that  a  portion  of  the  cost  was  for  upgrading  Eglin  range  capabilities 
(e.g.,  Time-Space-Position  Information  Data  Processor  [TDP]  improvements)  and  that  these 
costs  would  not  normally  be  borne  by  the  PGM  testing  program 

Cost  data  are  also  available  for  the  linked  laboratory  distributed  testing  configuration  as 
implemented  in  the  SIT  LSP.  In  comparing  to  LFP  costs,  it  should  be  remembered  that  the  LSP 
configuration  included  an  air  intercept  missile  (AIM)-9  Sidewinder,  rather  than  an  AIM- 120 
AMRAAM,  HWIL  laboratory. 

•  LSP  distributed  testing  mission:  about  $20,000  for  a  four-hour  mission  that  achieved  80  runs 
(~$250  per  run) 

•  LSP  distributed  testing  implementation  cost:  about  $1.2  million  including  the  cost  of 
modifying  Point  Mugu/China  Lake,  California,  facilities  and  integration  testing.  This  cost 
reflected  the  need  to  retrofit  the  existing  facilities  so  that  they  could  be  linked.  If  the 


53 


requirement  for  linking  was  planned  into  the  facilities  design,  it  is  believed  that  the  cost  of 
implementing  linking  would  have  been  significantly  lower 

A.3.2  Testing  Program  Cost  Example 

As  an  example  of  estimating  the  cost  of  a  testing  program  with  and  without  distributed  testing,  a 
live  fire  test  program  without  distributed  testing  is  compared  to  a  test  program  in  which  (1)  some 
of  the  live  tests  are  replaced  with  distributed  tests  and  (2)  a  distributed  testing  configuration  is 
used  to  rehearse  the  live  tests. 

For  the  specific  example,  the  AMRAAM  FOT&E[3A]  test  plan6  was  examined  to  determine  the 
maximum  number  of  launch  profiles  for  which  the  SIT  LFP  distributed  testing  architecture  could 
provide  valid  data.  Criteria  for  selecting  candidate  profiles  included  F-15  or  F-16  shooter,  no 
multiple  shooters  with  simultaneous  launches,  fighter  drone  targets,  resolved  targets  (if  multiple 
targets  were  involved),  no  target  beam  maneuvers  during  missile  intercept,  and  no  chaff  CM.  Out 
of  29  live  profiles,  three  met  the  screening  criteria  and  were  feasible  candidates  for  distributed 
testing  implementation.  In  other  words,  a  maximum  of  three  FOT&E[3A]  profiles  could  be 
performed  using  the  LFP  distributed  testing  configuration  rather  than  live  fire.  For  each  live  fire 
profile  replaced  with  a  distributed  testing  mission,  the  cost  savings  was  about  $550,000.  Thus, 
about  $1.65  million  could  be  saved  if  all  three  live  profiles  were  performed  with  distributed  testing 
missions.  (Note  that  even  though  each  distributed  testing  mission  can  include  15  passes,  it  was 
not  possible  to  combine  all  three  live  profiles  into  a  single  distributed  testing  mission.  This  was 
because  of  differences  in  shooters  and  other  key  scenario  features  in  the  three  profiles.) 

The  live  fire  profiles  in  FOT&E[3A]  were  rehearsed  using  captive-carry  missions.  Data  collected 
from  the  captive-carry  missions  were  normally  used  after  the  mission  as  inputs  into  an  AMRAAM 
DSM  for  the  purpose  of  verifying  the  launch  profiles  and  missile  flyouts.  If  this  was  done  using 
the  LFP  distributed  testing  configuration  instead,  the  Missile  Simulation  Laboratory  (MISILAB) 
HWIL  laboratory  would  provide  higher  fidelity  missile  flyout  results  during  the  mission  allowing 
real-time  verification  and  refinement  of  the  live  fire  profiles.  For  each  captive-carry  mission 
replaced  by  the  distributed  testing  configuration,  the  additional  cost  was  about  $20,000. 
Assuming  all  three  live  profiles  which  could  be  replaced  by  distributed  testing  were  indeed 
replaced  and  the  26  remaining  live  profiles  were  all  rehearsed  using  the  distributed  testing  LFP 
configuration,  the  additional  cost  for  the  rehearsals  would  be  about  $460,000  (three  profiles 
which  would  have  been  rehearsed  using  captive-carry  missions  were  eliminated,  saving  about 
$60,000),  and  the  net  cost  savings  from  reducing  the  number  of  live  shots  would  be  about  $1.2 
million. 

Note  that  the  nonrecurring  implementation  cost  for  the  LFP  distributed  testing  architecture  was 
about  $2  million.  When  this  investment  cost  is  also  considered,  the  maximum  use  of  distributed 
testing  in  support  of  FOT&E[3A]  would  have  cost  an  additional  $350,000  if  distributed  testing 


6  Richter,  G.  and  Volk,  P.,  AIM- 120  Advanced  Medium  Ranee  Air-to-Air  Missile  (AMRAAM) 
Follow-On  Onerafional  Test  and  Evaluation.  Phase  3.  Part  A  Test  Plan.  Air  Combat  Command, 
Nellis  AFB,  Nevada,  August  1996. 


54 


were  only  used  to  replace  three  of  the  live  profiles  and  an  additional  $870,000  if  distributed 
testing  were  also  used  to  rehearse  all  live  profiles. 

This  example  illustrates  that  if  distributed  testing  is  only  used  to  support  one  phase  of  testing 
involving  a  relatively  small  number  of  live  fire  tests,  cost  savings  may  not  be  possible.  If 
distributed  testing  is  instead  implemented  over  all  phases  of  testing,  cost  savings  would  be 
expected.  For  example,  the  analysis  of  the  FOT&E[3A]  profiles  showed  that  about  10%  of  them 
could  be  replaced  with  distributed  testing.  If  this  same  percentage  is  applied  to  a  total  missile 
testing  program  consisting  of  100  live  shots,  then  the  number  that  could  be  replaced  would  be  ten. 
In  this  case  (assuming  the  same  costs  as  for  AMRAAM  FOT&E[3A]  and  the  SIT  LFP),  there 
would  be  a  cost  savings  of  about  $1.9  million  (after  accounting  for  the  $2  million  distributed 
testing  implementation  cost)  if  the  distributed  testing  configuration  was  also  used  to  rehearse  all 
90  live  profiles  and  a  cost  savings  of  about  $3.7  million  if  the  distributed  testing  configuration  was 
not  used  for  rehearsal.  The  increased  cost  savings  with  each  replaced  live  shot  are  illustrated  in 
Figure  A2  (for  the  case  of  not  using  the  distributed  testing  configuration  for  rehearsal).  Note  that 
the  break-even  number  of  reduced  live  shots  is  four  (i.e.,  this  number  of  reduced  shots  is  required 
to  recover  the  $2  million  distributed  testing  development  cost). 

The  examples  in  this  section  are  meant  only  to  illustrate  the  application  of  the  cost  benefit  analysis 
methodology  and  should  not  be  considered  as  general  results.  Each  specific  PGM  program  must 
be  analyzed  to  determine  the  cost  effectiveness  of  distributed  testing  implementation.  In 
particular,  if  distributed  testing  implementation  is  properly  planned  for  early  in  the  acquisition  and 
testing  program,  the  cost  of  implementation  may  well  be  significantly  lower  than  the  $2  million 
distributed  testing  development  cost  in  these  examples.  (Note  that  implementation  costs  are 
indeed  strongly  dependent  on  the  specific  distributed  testing  architecture  —  the  SIT  LSP 
development  cost  was  about  $1.2  million  versus  about  $2  million  for  the  SIT  LFP.) 
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Figure  A2.  Levels  of  Program  Costs/Savings  With 
Varying  Numbers  of  Distributed  Testing  Replacement  Missions 

A.4.0  Conclusion 

A  general  methodology  has  been  presented  for  analyzing  the  cost  effectiveness  of  implementing 
distributed  testing  in  a  PGM  testing  program.  The  application  of  this  methodology  should  always 
be  done  in  a  total  program  sense  in  which  distributed  testing  is  blended  into  the  entire  PGM 
acquisition  and  testing  program.  If  the  strengths  and  benefits  of  distributed  testing  are  properly 
exploited,  it  should  be  possible  to  design  an  improved,  more  thorough,  and  higher  confidence  test 
and  evaluation  program,  while  still  remaining  within  budgetary  constraints. 
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Appendix  B 

Cost  Benefit  Analysis  for  the  End-to-End  Test 

A  study  performed  by  the  JADS  End-to-End  Test  team 

B.l  Follow-On  Full  Scale  Development  (FOFSD)  A-Tracker  Testing 

One  of  the  key  issues  surrounding  the  evaluation  of  the  utility  of  distributed  testing  is  the  cost 
benefit  of  using  it  to  augment  testing  of  a  command,  control,  communications,  computers, 
intelligence,  surveillance  and  reconnaissance  (C4ISR)  system.  The  JADS  End-to-End  (ETE)  Test 
team  chose  the  ETE  developmental  test  (DT)  as  the  most  appropriate  test  to  attempt  to  compare 
the  costs  and  benefits  of  this  type  of  testing  with  traditional  methodologies.  The  ETE  DT  was 
designed  as  a  series  of  contractor  investigative  tests  of  the  tracker  software  functions  of  the  E-8C 
operation  and  control  subsystem.  Each  of  the  tests  evaluated  the  performance  of  the  tracker 
software  against  a  ground  scenario  based  on  the  system  specifications  of  the  aircraft.  This  test 
was  compared  to  the  tracker  testing  conducted  by  the  contractor  and  the  Joint  Surveillance 
Target  Attack  Radar  System  (Joint  STARS)  Joint  Test  Force  (JTF)  during  the  FoFSD  phase  of 
the  contract.  This  report  will  identify  the  components  and  scope  of  the  FoFSD  tests,  the 
components  and  scope  of  the  ETE  DT  and  the  single  planned  flight  test  for  the  tracker  function. 
Costs  to  execute  the  test  are  reported  in  the  form  of  types  of  hours  used  to  conduct  the  tests. 
Actual  dollar  values  are  not  discussed.  Benefits  are  based  on  the  number  of  test  cases  supported 
in  each  test  and  the  limitations  of  each  test  methodology. 

B.2  Test  Requirements 

The  E-8C  tracker  is  a  software  function  that  allows  the  operator  to  initiate  and  track  a  group  of 
moving  targets  and  to  maintain  the  track  for  the  operator.  The  specifications  for  the  aircraft 
identify  the  performance  requirements  for  the  tracker  based  on  ground  operations.  A  test  case  for 
the  tracker  consisted  of  some  combination  of  the  following. 

Type  of  formations 
Convoy 
Block 

Line  abreast 
Number  of  vehicles 
Spacing  between  vehicles 
Speed  of  vehicles 
Type  of  maneuvers 
Turns 

Slowdown/speedup 
Stops  and  starts 
Cartographic 

The  combination  of  the  above  specifications  yielded  204  test  cases. 
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B.3  FoFSD  Testing 


During  FoFSD,  Northrop  Grumman  and  the  JTF  conducted  seven  dedicated  flight  tests  to  verify 
compliance  with  the  specifications.  Three  of  these  tests  were  contractor  developmental  test  and 
evaluation  (CDT&E)  and  four  of  the  tests  were  government-run  formal  qualification  test  flights. 
Each  of  the  tests  was  flown  over  the  range  at  Eglin  Air  Force  Base  (AFB),  Florida.  The  test  wing 
at  Eglin  AFB  supplied  instrumented  vehicles  and  drivers  to  the  test.  The  ability  to  test  each  of  the 
204  test  cases  was  limited  by  the  size  of  the  range  (3.5  total  miles  with  approximately  one  mile 
usable  for  testing)  and  range  safety  issues.  The  size  of  the  range  limited  the  number  of  vehicles 
used  in  a  test,  the  formations  that  could  be  used  and  the  type  of  maneuvers  that  could  be 
conducted.  Range  safety  limited  the  spacing  between  the  vehicles  and  the  speeds  that  could  be 
used.  The  following  is  a  review  of  the  data  on  these  flights. 


No 

Type  of 
Flight 

Flight 

Hours 

A/C 

Personnel 

A/C 

Personnel 

Required 

Drivers 

and 

Observers 

Test  Cases 

1 

CDT&E 

22 

7.7 

25-29 

10 

5 

N/A 

2 

CDT&E 

27 

7.1 

25-29 

10 

5 

N/A 

3 

CDT&E 

39 

7.2 

25-29 

5 

N/A 

4 

QT&E 

N/A 

6.9 

25-29 

ESM 

5 

N/A 

5 

QT&E 

198 

7.2 

26 

8 

26-27* 

3 

6 

206 

5.8 

24 

9 

15 

2 

wm 

222 

6.9 

26 

9 

26-27* 

3 

A/C  =  aircraft  N/A  -  information  not  available  QT&E  =qualification  test  and  evaluation 


*  Flights  198  and  222  included  a  light  aircraft  as  a  ground  target 

The  following  is  a  summary  of  the  cost  benefit  factors  of  the  previous  flight  testing. 


Number  of  test  flights 

7 

Length  of  test  flights 

5.8  hours  -  7.7  hours 

Number  of  personnel  in  the  aircraft 

24-29 

Number  of  vehicles  on  the  ground 

4-21  (one  light  aircraft  on  two  flights) 

Number  of  drivers  and  observers  on  the  ground 

8-27 

Test  cases  performed  per  flight 

2-3 

B.4  Live  Fly  Test  Limitations 

The  Eglin  AFB  test  range  was  the  most  limiting  factor  to  conducting  live  tracker  tests.  The  3.5- 
mile  test  run  only  provided  approximately  1800  meters  of  usable  test  run.  This  is  due  to  the 
length  of  time  required  to  get  the  entire  group  of  vehicles  up  to  speed  and  at  the  correct  spacing 
and  the  length  of  time  required  for  them  to  slowdown  and  stop.  This  limitation  impacts  the 
number  of  vehicles  that  can  be  part  of  a  convoy.  The  width  of  the  range  also  limits  the  number  of 
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vehicles  that  can  be  in  a  line  abreast  formation.  The  width  also  denies  the  ability  to  test  block 
formations. 

Range  safety  issues  require  slower  speeds  and  wider  spacing  than  the  specifications  required. 
Likewise,  the  relative  inexperience  of  the  drivers  resulted  in  their  inability  to  maintain  spacing 
during  maneuvers.  Currently,  Eglin  AFB  is  limited  to  fielding  20  ground  positioning  system 
(GPS)-equipped  vehicles.  For  one  test,  Eglin  provided  20  GPS-equipped  vehicles  and  10 
additional  vehicles  equipped  with  another  time-space-position  information  (TSPI)  system; 
however,  this  test  was  the  most  expensive  test  that  had  been  tried  and  will  probably  not  be 
repeated.  Because  of  costs,  most  tests  are  limited  to  10  or  fewer  ground  vehicles.  Repeatability 
of  these  tests  was  limited  by  both  driver  accuracy  and  by  weather  conditions  at  Eglin  AFB. 

B.5  Distributed  Testing 

Distributed  testing  uses  a  combination  of  a  rigorous  test  of  the  tracker  using  a  distributed  test 
environment  and  a  single  test  flight  to  verify  continued  performance  of  the  tracker.  The 
increasing  cost  of  flight  tests  has  forced  the  Air  Force  to  reduce  the  number  of  flights  available  for 
testing  upgrades  and  for  regression  testing.  The  tracker  investigations  are  being  conducted  as 
part  of  the  tactical  digital  information  link  (TADIL)  J  upgrade  (TJU)  program  for  the  E-8C.  Prior 
to  considering  distributed  testing,  the  JTF  had  already  accepted  the  need  to  reduce  testing  to  a 
single  flight.  The  cooperative  effort  between  JADS,  Grumman,  and  the  JTF  provided  an 
opportunity  for  expanded  testing  of  the  tracker  while  allowing  JADS  to  evaluate  the  utility  of 
distributed  testing  to  a  development  test. 

The  distributed  testing  environment  for  the  tracker  test  consisted  of  a  scenario  generation  and 
transmission  capability  at  the  JADS  Test  Control  and  Analysis  Center  (TCAC),  a  scenario  logger 
and  playback  capability  in  the  JADS  laboratory  at  Grumman,  and  a  version  of  the  JADS- 
developed  Virtual  Surveillance  Target  Attack  Radar  System  (VSTARS)  located  in  the  Operation 
and  Control  Test  Laboratory  at  Grumman.  The  distributed  testing  synthetic  test  range  was  laid 
out  using  a  Southwest  Asia  terrain  file.  Terrain  features  had  been  modified  to  increase  the 
movement  fidelity. 

Grumman  analyzed  the  tracker  system  requirements  and  produced  a  set  of  requirements  for 
vehicles  and  their  movements.  JADS  ETE  Test  personnel  translated  these  requirements  into 
constructive  simulations  of  all  204  test  cases  implemented  using  the  Janus  6K  scenario  driver. 
These  simulations  provided  high-fidelity  movement  of  the  vehicles  using  12  scenarios:  accuracy; 
continuity  1,  2,  and  3;  block  formation  1,  2,  and  3;  line  abreast  1,  2,  and  3;  in-line  traffic  and 
cartographic. 

Once  the  scenarios  were  developed  and  documented,  they  were  transmitted  from  the  TCAC 
facility  to  the  logger  at  Grumman.  During  these  downloads,  data  were  collected  for  further 
analysis  of  latency  and  protocol  data  unit  (PDU)  dropout  rates. 

After  the  scenarios  were  logged  at  Grumman,  the  Grumman  logger  was  switched  to  an  internal 
local  area  network  for  connection  to  a  version  of  VSTARS  that  had  been  integrated  with  the  TJU 
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build  of  software.  Each  scenario  was  played  and  the  test  engineers/operators  established  tracks 
and  collected  data.  The  scenarios  could  be  replayed  to  obtain  greater  numbers  of  data  samples. 

The  following  summarizes  the  cost  benefit  factors  of  the  distributed  testing. 


Number  of  test  flights 

1 

Length  of  test  flight  (estimated) 

7  hours 

Number  of  personnel  in  the  aircraft 

10-29 

Number  of  vehicles  on  the  ground  4-21 

4-21 

Number  of  drivers  and  observers  on  the  ground 

8-27 

Test  cases  performed  during  flight 

2-3 

Number  of  distributed  testing  sessions 

12 

Total  length  of  test  sessions 

27  days 

Number  of  personnel  conducting  test 

2-3 

Test  cases  performed  during  distributed  testing 

204 

B.6  Conclusions 


Inadequate  range  facilities,  range  safety  issues,  and  insufficient  numbers  of  and  the  high  costs 
associated  with  the  test  assets  limit  traditional  means  of  testing  C4ISR  systems,  especially  those 
using  sensor  systems  that  cover  a  large  area  of  the  battlefield,  such  as  Joint  STARS.  As  this  cost 
benefit  example  illustrates,  distributed  testing  technology  can  eliminate  many  of  these 
conventional  testing  limitations  Using  distributed  testing,  testers  can  cost  effectively  conduct 
more  test  trials  for  longer  periods  of  time.  For  example  during  the  Phase  2  test,  the  ETE  Test 
team  was  able  to  conduct  five  test  trials,  lasting  six  hours  each,  within  a  5-day  period.  A 
maximum  of  three  trials  could  have  been  conducted  using  the  test  aircraft.  If  additional  shifts  of 
operators  were  available,  the  test  trails  could  have  lasted  longer  at  little  additional  cost.  Because 
of  scheduling  issues,  crew  rest,  and  the  lack  of  test  aircraft,  it  would  be  difficult  to  extend  a  live 
test  trial  beyond  five  to  six  hours  on  station.  Comparison  of  the  cost  benefit  factors  of  the 
previous  FoFSD  live  flight  and  the  cost  benefit  factors  of  distributed  testing  shows  that  distributed 
testing  can  realistically  test  C4ISR  systems: 

-  in  larger  battle  spaces 

-  with  larger  numbers  of  ground-based  entities 

-  under  realistic,  but  unsafe  conditions 

-  using  multiple  repetitions  of  the  same  test  scenario  without  competing  for  scarce  (i.e. 
expensive)  test  aircraft  and  range  resources 


61 


